system/freebsd-src
system/freebsd-src
1
0
Fork 0
mirror of https://github.com/freebsd/freebsd-src synced 2024-10-06 16:40:47 +00:00
Code Issues Packages Projects Releases Wiki Activity

Vendor import of xz 5.6.2 (trimmed)

Browse source
This commit is contained in:
Xin LI 2024-06-02 22:49:22 -07:00
parent a7819ed7ad
commit b599c3410c
51 changed files with 3508 additions and 3690 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
AUTHORS
View file

@ -2,8 +2,8 @@
Authors of XZ Utils
===================
XZ Utils is developed and maintained by Lasse Collin
<lasse.collin@tukaani.org> and Jia Tan <jiat0218@gmail.com>.
XZ Utils is developed and maintained by
Lasse Collin <lasse.collin@tukaani.org>.
Major parts of liblzma are based on code written by Igor Pavlov,
specifically the LZMA SDK <https://7-zip.org/sdk.html>. Without
@ -30,6 +30,11 @@ Authors of XZ Utils
- Jonathan Nieder
- Joachim Henke
Special author: Jia Tan was a co-maintainer in 2022-2024. He and
the team behind him inserted a backdoor (CVE-2024-3094) into
XZ Utils 5.6.0 and 5.6.1 releases. He suddenly disappeared when
this was discovered.
Many people have contributed improvements or reported bugs.
Most of these people are mentioned in the file THANKS.

27
COPYING
View file

@ -3,8 +3,7 @@ XZ Utils Licensing
==================
Different licenses apply to different files in this package. Here
is a rough summary of which licenses apply to which parts of this
package (but check the individual files to be sure!):
is a summary of which licenses apply to which parts of this package:
- liblzma is under the BSD Zero Clause License (0BSD).
@ -25,20 +24,6 @@ XZ Utils Licensing
lzma-file-format.xt are in the public domain but may
be distributed under the terms of 0BSD too.
- Doxygen-generated HTML version of the liblzma API documentation:
While Doxygen is under the GNU GPLv2, the license information
in Doxygen includes the following exception:
Documents produced by doxygen are derivative works
derived from the input used in their production;
they are not affected by this license.
Note: The JavaScript files (under the MIT license) have
been removed from the Doxygen output.
- The XZ logo (xz-logo.png) included in the Doxygen-generated
documentation is under the Creative Commons BY-SA 4.0 license.
- Translated messages and man pages are under 0BSD except that
some old translations are in the public domain.
@ -51,8 +36,9 @@ XZ Utils Licensing
These files don't affect the licensing of the binaries being
built.
- The extra directory contain files that are under various
free software licenses.
- The 'extra' directory contains files that are under various
free software licenses. These aren't built or installed as
part of XZ Utils.
For the files under the BSD Zero Clause License (0BSD), if
a copyright notice is needed, the following is sufficient:
@ -65,16 +51,13 @@ XZ Utils Licensing
it is not legally required by the license terms. Here is an example
of a good notice to put into "about box" or into documentation:
This software includes code from XZ Utils
<https://xz.tukaani.org/xz-utils/>.
This software includes code from XZ Utils <https://tukaani.org/xz/>.
The following license texts are included in the following files:
- COPYING.0BSD: BSD Zero Clause License
- COPYING.LGPLv2.1: GNU Lesser General Public License version 2.1
- COPYING.GPLv2: GNU General Public License version 2
- COPYING.GPLv3: GNU General Public License version 3
- COPYING.CC-BY-SA-4.0: Creative Commons Attribution-ShareAlike 4.0
International Public License
A note about old XZ Utils releases:

427
COPYING.CC-BY-SA-4.0
View file

@ -1,427 +0,0 @@
Attribution-ShareAlike 4.0 International
=======================================================================
Creative Commons Corporation ("Creative Commons") is not a law firm and
does not provide legal services or legal advice. Distribution of
Creative Commons public licenses does not create a lawyer-client or
other relationship. Creative Commons makes its licenses and related
information available on an "as-is" basis. Creative Commons gives no
warranties regarding its licenses, any material licensed under their
terms and conditions, or any related information. Creative Commons
disclaims all liability for damages resulting from their use to the
fullest extent possible.
Using Creative Commons Public Licenses
Creative Commons public licenses provide a standard set of terms and
conditions that creators and other rights holders may use to share
original works of authorship and other material subject to copyright
and certain other rights specified in the public license below. The
following considerations are for informational purposes only, are not
exhaustive, and do not form part of our licenses.
Considerations for licensors: Our public licenses are
intended for use by those authorized to give the public
permission to use material in ways otherwise restricted by
copyright and certain other rights. Our licenses are
irrevocable. Licensors should read and understand the terms
and conditions of the license they choose before applying it.
Licensors should also secure all rights necessary before
applying our licenses so that the public can reuse the
material as expected. Licensors should clearly mark any
material not subject to the license. This includes other CC-
licensed material, or material used under an exception or
limitation to copyright. More considerations for licensors:
wiki.creativecommons.org/Considerations_for_licensors
Considerations for the public: By using one of our public
licenses, a licensor grants the public permission to use the
licensed material under specified terms and conditions. If
the licensor's permission is not necessary for any reason--for
example, because of any applicable exception or limitation to
copyright--then that use is not regulated by the license. Our
licenses grant only permissions under copyright and certain
other rights that a licensor has authority to grant. Use of
the licensed material may still be restricted for other
reasons, including because others have copyright or other
rights in the material. A licensor may make special requests,
such as asking that all changes be marked or described.
Although not required by our licenses, you are encouraged to
respect those requests where reasonable. More considerations
for the public:
wiki.creativecommons.org/Considerations_for_licensees
=======================================================================
Creative Commons Attribution-ShareAlike 4.0 International Public
License
By exercising the Licensed Rights (defined below), You accept and agree
to be bound by the terms and conditions of this Creative Commons
Attribution-ShareAlike 4.0 International Public License ("Public
License"). To the extent this Public License may be interpreted as a
contract, You are granted the Licensed Rights in consideration of Your
acceptance of these terms and conditions, and the Licensor grants You
such rights in consideration of benefits the Licensor receives from
making the Licensed Material available under these terms and
conditions.
Section 1 -- Definitions.
a. Adapted Material means material subject to Copyright and Similar
Rights that is derived from or based upon the Licensed Material
and in which the Licensed Material is translated, altered,
arranged, transformed, or otherwise modified in a manner requiring
permission under the Copyright and Similar Rights held by the
Licensor. For purposes of this Public License, where the Licensed
Material is a musical work, performance, or sound recording,
Adapted Material is always produced where the Licensed Material is
synched in timed relation with a moving image.
b. Adapter's License means the license You apply to Your Copyright
and Similar Rights in Your contributions to Adapted Material in
accordance with the terms and conditions of this Public License.
c. BY-SA Compatible License means a license listed at
creativecommons.org/compatiblelicenses, approved by Creative
Commons as essentially the equivalent of this Public License.
d. Copyright and Similar Rights means copyright and/or similar rights
closely related to copyright including, without limitation,
performance, broadcast, sound recording, and Sui Generis Database
Rights, without regard to how the rights are labeled or
categorized. For purposes of this Public License, the rights
specified in Section 2(b)(1)-(2) are not Copyright and Similar
Rights.
e. Effective Technological Measures means those measures that, in the
absence of proper authority, may not be circumvented under laws
fulfilling obligations under Article 11 of the WIPO Copyright
Treaty adopted on December 20, 1996, and/or similar international
agreements.
f. Exceptions and Limitations means fair use, fair dealing, and/or
any other exception or limitation to Copyright and Similar Rights
that applies to Your use of the Licensed Material.
g. License Elements means the license attributes listed in the name
of a Creative Commons Public License. The License Elements of this
Public License are Attribution and ShareAlike.
h. Licensed Material means the artistic or literary work, database,
or other material to which the Licensor applied this Public
License.
i. Licensed Rights means the rights granted to You subject to the
terms and conditions of this Public License, which are limited to
all Copyright and Similar Rights that apply to Your use of the
Licensed Material and that the Licensor has authority to license.
j. Licensor means the individual(s) or entity(ies) granting rights
under this Public License.
k. Share means to provide material to the public by any means or
process that requires permission under the Licensed Rights, such
as reproduction, public display, public performance, distribution,
dissemination, communication, or importation, and to make material
available to the public including in ways that members of the
public may access the material from a place and at a time
individually chosen by them.
l. Sui Generis Database Rights means rights other than copyright
resulting from Directive 96/9/EC of the European Parliament and of
the Council of 11 March 1996 on the legal protection of databases,
as amended and/or succeeded, as well as other essentially
equivalent rights anywhere in the world.
m. You means the individual or entity exercising the Licensed Rights
under this Public License. Your has a corresponding meaning.
Section 2 -- Scope.
a. License grant.
1. Subject to the terms and conditions of this Public License,
the Licensor hereby grants You a worldwide, royalty-free,
non-sublicensable, non-exclusive, irrevocable license to
exercise the Licensed Rights in the Licensed Material to:
a. reproduce and Share the Licensed Material, in whole or
in part; and
b. produce, reproduce, and Share Adapted Material.
2. Exceptions and Limitations. For the avoidance of doubt, where
Exceptions and Limitations apply to Your use, this Public
License does not apply, and You do not need to comply with
its terms and conditions.
3. Term. The term of this Public License is specified in Section
6(a).
4. Media and formats; technical modifications allowed. The
Licensor authorizes You to exercise the Licensed Rights in
all media and formats whether now known or hereafter created,
and to make technical modifications necessary to do so. The
Licensor waives and/or agrees not to assert any right or
authority to forbid You from making technical modifications
necessary to exercise the Licensed Rights, including
technical modifications necessary to circumvent Effective
Technological Measures. For purposes of this Public License,
simply making modifications authorized by this Section 2(a)
(4) never produces Adapted Material.
5. Downstream recipients.
a. Offer from the Licensor -- Licensed Material. Every
recipient of the Licensed Material automatically
receives an offer from the Licensor to exercise the
Licensed Rights under the terms and conditions of this
Public License.
b. Additional offer from the Licensor -- Adapted Material.
Every recipient of Adapted Material from You
automatically receives an offer from the Licensor to
exercise the Licensed Rights in the Adapted Material
under the conditions of the Adapter's License You apply.
c. No downstream restrictions. You may not offer or impose
any additional or different terms or conditions on, or
apply any Effective Technological Measures to, the
Licensed Material if doing so restricts exercise of the
Licensed Rights by any recipient of the Licensed
Material.
6. No endorsement. Nothing in this Public License constitutes or
may be construed as permission to assert or imply that You
are, or that Your use of the Licensed Material is, connected
with, or sponsored, endorsed, or granted official status by,
the Licensor or others designated to receive attribution as
provided in Section 3(a)(1)(A)(i).
b. Other rights.
1. Moral rights, such as the right of integrity, are not
licensed under this Public License, nor are publicity,
privacy, and/or other similar personality rights; however, to
the extent possible, the Licensor waives and/or agrees not to
assert any such rights held by the Licensor to the limited
extent necessary to allow You to exercise the Licensed
Rights, but not otherwise.
2. Patent and trademark rights are not licensed under this
Public License.
3. To the extent possible, the Licensor waives any right to
collect royalties from You for the exercise of the Licensed
Rights, whether directly or through a collecting society
under any voluntary or waivable statutory or compulsory
licensing scheme. In all other cases the Licensor expressly
reserves any right to collect such royalties.
Section 3 -- License Conditions.
Your exercise of the Licensed Rights is expressly made subject to the
following conditions.
a. Attribution.
1. If You Share the Licensed Material (including in modified
form), You must:
a. retain the following if it is supplied by the Licensor
with the Licensed Material:
i. identification of the creator(s) of the Licensed
Material and any others designated to receive
attribution, in any reasonable manner requested by
the Licensor (including by pseudonym if
designated);
ii. a copyright notice;
iii. a notice that refers to this Public License;
iv. a notice that refers to the disclaimer of
warranties;
v. a URI or hyperlink to the Licensed Material to the
extent reasonably practicable;
b. indicate if You modified the Licensed Material and
retain an indication of any previous modifications; and
c. indicate the Licensed Material is licensed under this
Public License, and include the text of, or the URI or
hyperlink to, this Public License.
2. You may satisfy the conditions in Section 3(a)(1) in any
reasonable manner based on the medium, means, and context in
which You Share the Licensed Material. For example, it may be
reasonable to satisfy the conditions by providing a URI or
hyperlink to a resource that includes the required
information.
3. If requested by the Licensor, You must remove any of the
information required by Section 3(a)(1)(A) to the extent
reasonably practicable.
b. ShareAlike.
In addition to the conditions in Section 3(a), if You Share
Adapted Material You produce, the following conditions also apply.
1. The Adapter's License You apply must be a Creative Commons
license with the same License Elements, this version or
later, or a BY-SA Compatible License.
2. You must include the text of, or the URI or hyperlink to, the
Adapter's License You apply. You may satisfy this condition
in any reasonable manner based on the medium, means, and
context in which You Share Adapted Material.
3. You may not offer or impose any additional or different terms
or conditions on, or apply any Effective Technological
Measures to, Adapted Material that restrict exercise of the
rights granted under the Adapter's License You apply.
Section 4 -- Sui Generis Database Rights.
Where the Licensed Rights include Sui Generis Database Rights that
apply to Your use of the Licensed Material:
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
to extract, reuse, reproduce, and Share all or a substantial
portion of the contents of the database;
b. if You include all or a substantial portion of the database
contents in a database in which You have Sui Generis Database
Rights, then the database in which You have Sui Generis Database
Rights (but not its individual contents) is Adapted Material,
including for purposes of Section 3(b); and
c. You must comply with the conditions in Section 3(a) if You Share
all or a substantial portion of the contents of the database.
For the avoidance of doubt, this Section 4 supplements and does not
replace Your obligations under this Public License where the Licensed
Rights include other Copyright and Similar Rights.
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
c. The disclaimer of warranties and limitation of liability provided
above shall be interpreted in a manner that, to the extent
possible, most closely approximates an absolute disclaimer and
waiver of all liability.
Section 6 -- Term and Termination.
a. This Public License applies for the term of the Copyright and
Similar Rights licensed here. However, if You fail to comply with
this Public License, then Your rights under this Public License
terminate automatically.
b. Where Your right to use the Licensed Material has terminated under
Section 6(a), it reinstates:
1. automatically as of the date the violation is cured, provided
it is cured within 30 days of Your discovery of the
violation; or
2. upon express reinstatement by the Licensor.
For the avoidance of doubt, this Section 6(b) does not affect any
right the Licensor may have to seek remedies for Your violations
of this Public License.
c. For the avoidance of doubt, the Licensor may also offer the
Licensed Material under separate terms or conditions or stop
distributing the Licensed Material at any time; however, doing so
will not terminate this Public License.
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
License.
Section 7 -- Other Terms and Conditions.
a. The Licensor shall not be bound by any additional or different
terms or conditions communicated by You unless expressly agreed.
b. Any arrangements, understandings, or agreements regarding the
Licensed Material not stated herein are separate from and
independent of the terms and conditions of this Public License.
Section 8 -- Interpretation.
a. For the avoidance of doubt, this Public License does not, and
shall not be interpreted to, reduce, limit, restrict, or impose
conditions on any use of the Licensed Material that could lawfully
be made without permission under this Public License.
b. To the extent possible, if any provision of this Public License is
deemed unenforceable, it shall be automatically reformed to the
minimum extent necessary to make it enforceable. If the provision
cannot be reformed, it shall be severed from this Public License
without affecting the enforceability of the remaining terms and
conditions.
c. No term or condition of this Public License will be waived and no
failure to comply consented to unless expressly agreed to by the
Licensor.
d. Nothing in this Public License constitutes or may be interpreted
as a limitation upon, or waiver of, any privileges and immunities
that apply to the Licensor or You, including from the legal
processes of any jurisdiction or authority.
=======================================================================
Creative Commons is not a party to its public
licenses. Notwithstanding, Creative Commons may elect to apply one of
its public licenses to material it publishes and in those instances
will be considered the “Licensor.” The text of the Creative Commons
public licenses is dedicated to the public domain under the CC0 Public
Domain Dedication. Except for the limited purpose of indicating that
material is shared under a Creative Commons public license or as
otherwise permitted by the Creative Commons policies published at
creativecommons.org/policies, Creative Commons does not authorize the
use of the trademark "Creative Commons" or any other trademark or logo
of Creative Commons without its prior written consent including,
without limitation, in connection with any unauthorized modifications
to any of its public licenses or any other arrangements,
understandings, or agreements concerning use of licensed material. For
the avoidance of doubt, this paragraph does not form part of the
public licenses.
Creative Commons may be contacted at creativecommons.org.

2898
ChangeLog
View file

File diff suppressed because it is too large Load diff

26
README
View file

@ -79,8 +79,6 @@ XZ Utils
COPYING.GPLv2 GNU General Public License version 2
COPYING.GPLv3 GNU General Public License version 3
COPYING.LGPLv2.1 GNU Lesser General Public License version 2.1
COPYING.CC-BY-SA-4.0 Creative Commons Attribution-ShareAlike 4.0
International Public License
AUTHORS The main authors of XZ Utils
THANKS Incomplete list of people who have helped making
@ -97,9 +95,9 @@ XZ Utils
The command-line tools are documented as man pages. In source code
releases (and possibly also in some binary packages), the man pages
are also provided in plain text (ASCII only) and PDF formats in the
directory "doc/man" to make the man pages more accessible to those
whose operating system doesn't provide an easy way to view man pages.
are also provided in plain text (ASCII only) format in the directory
"doc/man" to make the man pages more accessible to those whose
operating system doesn't provide an easy way to view man pages.
1.3. Documentation for liblzma
@ -290,21 +288,23 @@ XZ Utils
XZ Embedded is a limited implementation written for use in the Linux
kernel, but it is also suitable for other embedded use.
https://xz.tukaani.org/xz-embedded/
https://tukaani.org/xz/embedded.html
XZ for Java is a complete implementation written in pure Java.
https://xz.tukaani.org/xz-for-java/
https://tukaani.org/xz/java.html
6. Contact information
----------------------
If you have questions, bug reports, patches etc. related to XZ Utils,
the project maintainers Lasse Collin and Jia Tan can be reached via
<xz@tukaani.org>.
XZ Utils in general:
- Home page: https://tukaani.org/xz/
- Email to maintainer(s): xz@tukaani.org
- IRC: #tukaani on Libera Chat
- GitHub: https://github.com/tukaani-project/xz
You might find Lasse also from #tukaani on Libera Chat (IRC).
The nick is Larhzu. The channel tends to be pretty quiet,
so just ask your question and someone might wake up.
Lead maintainer:
- Email: Lasse Collin <lasse.collin@tukaani.org>
- IRC: Larhzu on Libera Chat

6
THANKS
View file

@ -46,6 +46,7 @@ has been important. :-) In alphabetical order:
- Vincent Fazio
- Michael Felt
- Michael Fox
- Andres Freund
- Mike Frysinger
- Daniel Richard G.
- Tomasz Gajc
@ -76,6 +77,7 @@ has been important. :-) In alphabetical order:
- Richard Koch
- Anton Kochkov
- Ville Koskinen
- Sergey Kosukhin
- Marcin Kowalczyk
- Jan Kratochvil
- Christian Kujau
@ -150,11 +152,10 @@ has been important. :-) In alphabetical order:
- Martin Storsjö
- Jonathan Stott
- Dan Stromberg
- Jia Tan
- Vincent Torri
- Alexey Tourbin
- Paul Townsend
- Mohammed Adnène Trojette
- Alexey Tourbin
- Taiki Tsunekawa
- Maksym Vatsyk
- Loganaden Velvindron
@ -171,6 +172,7 @@ has been important. :-) In alphabetical order:
- Charles Wilson
- Lars Wirzenius
- Pilorz Wojciech
- Chien Wong
- Ryan Young
- Andreas Zieringer

2672
doxygen/Doxyfile
View file

File diff suppressed because it is too large Load diff

13
doxygen/footer.html
View file

@ -1,13 +0,0 @@
<hr class="footer"/>
<p style="text-align: right;padding-right: 12px;">
XZ logo &copy; 2023 by Jia Tan is licensed under
<a href="COPYING.CC-BY-SA-4.0"
rel="license"
style="display:inline-block;">
CC BY-SA 4.0
</a>
</p>
</body>
</html>

131
doxygen/update-doxygen
View file

@ -3,14 +3,13 @@
#############################################################################
#
# Updates the Doxygen generated documentation files in the source tree.
# If the doxygen command is not installed, it will exit with an error.
# This script can generate Doxygen documentation for all source files or for
# just liblzma API header files.
# While it's possible to use the Doxyfile as is to generate liblzma API
# documentation, it is recommended to use this script because this adds
# the XZ Utils version number to the generated HTML.
#
# It is recommended to use this script to update the Doxygen-generated HTML
# files since this will include the package version in the output and,
# in case of liblzma API docs, strip JavaScript files from the output.
# Other features:
# - Generate documentation of the XZ Utils internals.
# - Set input and output paths for out-of-tree builds.
#
#############################################################################
#
@ -21,31 +20,67 @@
set -e
show_usage()
{
echo "Usage: $0 <api|internal> [ABS_TOP_SRCDIR ABS_OUTDIR]"
echo
echo "Supported modes:"
echo " - 'api' (default): liblzma API docs into doc/api"
echo " - 'internal': internal docs into doc/internal"
echo
echo "Absolute source and output dirs may be set" \
"to do an out-of-tree build."
echo "The output directory must already exist."
exit 1
}
case $1 in
api|internal)
;;
*)
show_usage
;;
esac
if type doxygen > /dev/null 2>&1; then
:
else
echo "doxygen/update-doxygen: 'doxygen' command not found." >&2
echo "doxygen/update-doxygen: Skipping Doxygen docs generation." >&2
echo "$0: 'doxygen' command not found" >&2
exit 1
fi
if test ! -f Doxyfile; then
cd `dirname "$0"` || exit 1
if test ! -f Doxyfile; then
echo "doxygen/update-doxygen: Cannot find Doxyfile" >&2
exit 1
fi
case $# in
1)
# One argument: Building inside the source tree
ABS_TOP_SRCDIR=`dirname "$0"`/..
ABS_OUTDIR=$ABS_TOP_SRCDIR/doc
;;
3)
# Three arguments: Possibly an out of tree build
ABS_TOP_SRCDIR=$2
ABS_OUTDIR=$3
;;
*)
show_usage
;;
esac
if test ! -f "$ABS_TOP_SRCDIR/doxygen/Doxyfile"; then
echo "$0: Source dir '$ABS_TOP_SRCDIR/doxygen/Doxyfile' not found" >&2
exit 1
fi
if test ! -d "$ABS_OUTDIR"; then
echo "$0: Output dir '$ABS_OUTDIR' not found" >&2
exit 1
fi
# Get the package version so that it can be included in the generated docs.
PACKAGE_VERSION=`cd .. && sh build-aux/version.sh` || exit 1
PACKAGE_VERSION=`cd "$ABS_TOP_SRCDIR" && sh build-aux/version.sh`
# If no arguments are specified, default to generating liblzma API header
# documentation only.
case $1 in
'' | api)
api)
# Remove old documentation before re-generating the new.
rm -rf ../doc/api
rm -rf "$ABS_OUTDIR/api"
# Generate the HTML documentation by preparing the Doxyfile
# in stdin and piping the result to the doxygen command.
@ -53,57 +88,27 @@ case $1 in
# override any earlier assignment. So, we can use this
# feature to override the tags that need to change between
# "api" and "internal" modes.
ABS_SRCDIR=$ABS_TOP_SRCDIR/src/liblzma/api
(
cat Doxyfile
cat "$ABS_TOP_SRCDIR/doxygen/Doxyfile"
echo "PROJECT_NUMBER = $PACKAGE_VERSION"
) | doxygen -
# As of Doxygen 1.8.0 - 1.9.6 and the Doxyfile options we use,
# the output is good without any JavaScript. Unfortunately
# Doxygen doesn't have an option to disable JavaScript usage
# completely so we strip it away with the hack below.
#
# Omitting the JavaScript code avoids some license hassle
# as jquery.js is fairly big, it contains more than jQuery
# itself, and doesn't include the actual license text (it
# only refers to the MIT license by name).
echo "Stripping JavaScript from Doxygen output..."
for F in ../doc/api/*.html
do
sed 's/<script [^>]*><\/script>//g
s/onclick="[^"]*"//g' \
"$F" > ../doc/api/tmp
mv -f ../doc/api/tmp "$F"
done
rm -f ../doc/api/*.js
echo "OUTPUT_DIRECTORY = $ABS_OUTDIR"
echo "STRIP_FROM_PATH = $ABS_SRCDIR"
echo "INPUT = $ABS_SRCDIR"
) | doxygen -q -
;;
internal)
# The docs from internal aren't for distribution so
# the JavaScript files aren't an issue here.
rm -rf ../doc/internal
rm -rf "$ABS_OUTDIR/internal"
(
cat Doxyfile
echo "PROJECT_NUMBER = $PACKAGE_VERSION"
cat "$ABS_TOP_SRCDIR/doxygen/Doxyfile"
echo 'PROJECT_NAME = "XZ Utils"'
echo 'STRIP_FROM_PATH = ../src'
echo 'INPUT = ../src'
echo "PROJECT_NUMBER = $PACKAGE_VERSION"
echo "OUTPUT_DIRECTORY = $ABS_OUTDIR"
echo "STRIP_FROM_PATH = $ABS_TOP_SRCDIR"
echo "INPUT = $ABS_TOP_SRCDIR/src"
echo 'HTML_OUTPUT = internal'
echo 'EXTRACT_PRIVATE = YES'
echo 'EXTRACT_STATIC = YES'
echo 'EXTRACT_LOCAL_CLASSES = YES'
echo 'SEARCHENGINE = YES'
) | doxygen -
;;
*)
echo "doxygen/update-doxygen: Error: mode argument '$1'" \
"is not supported." >&2
echo "doxygen/update-doxygen: Supported modes:" >&2
echo "doxygen/update-doxygen: - 'api' (default):" \
"liblzma API docs into doc/api" >&2
echo "doxygen/update-doxygen: - 'internal':"\
"internal docs into doc/internal" >&2
exit 1
) | doxygen -q -
;;
esac

2
src/common/mythread.h
View file

@ -200,7 +200,7 @@ mythread_join(mythread thread)
}
// Initiatlizes a mutex. Returns zero on success and non-zero on error.
// Initializes a mutex. Returns zero on success and non-zero on error.
static inline int
mythread_mutex_init(mythread_mutex *mutex)
{

49
src/common/tuklib_integer.h
View file

@ -16,7 +16,7 @@
///
/// Endianness-converting integer operations (these can be macros!)
/// (XX = 16, 32, or 64; Y = b or l):
/// - Byte swapping: bswapXX(num)
/// - Byte swapping: byteswapXX(num)
/// - Byte order conversions to/from native (byteswaps if Y isn't
/// the native endianness): convXXYe(num)
/// - Unaligned reads: readXXYe(ptr)
@ -66,38 +66,41 @@
#if defined(HAVE___BUILTIN_BSWAPXX)
// GCC >= 4.8 and Clang
# define bswap16(n) __builtin_bswap16(n)
# define bswap32(n) __builtin_bswap32(n)
# define bswap64(n) __builtin_bswap64(n)
# define byteswap16(num) __builtin_bswap16(num)
# define byteswap32(num) __builtin_bswap32(num)
# define byteswap64(num) __builtin_bswap64(num)
#elif defined(HAVE_BYTESWAP_H)
// glibc, uClibc, dietlibc
# include <byteswap.h>
# ifdef HAVE_BSWAP_16
# define bswap16(num) bswap_16(num)
# define byteswap16(num) bswap_16(num)
# endif
# ifdef HAVE_BSWAP_32
# define bswap32(num) bswap_32(num)
# define byteswap32(num) bswap_32(num)
# endif
# ifdef HAVE_BSWAP_64
# define bswap64(num) bswap_64(num)
# define byteswap64(num) bswap_64(num)
# endif
#elif defined(HAVE_SYS_ENDIAN_H)
// *BSDs and Darwin
# include <sys/endian.h>
# define byteswap16(num) bswap16(num)
# define byteswap32(num) bswap32(num)
# define byteswap64(num) bswap64(num)
#elif defined(HAVE_SYS_BYTEORDER_H)
// Solaris
# include <sys/byteorder.h>
# ifdef BSWAP_16
# define bswap16(num) BSWAP_16(num)
# define byteswap16(num) BSWAP_16(num)
# endif
# ifdef BSWAP_32
# define bswap32(num) BSWAP_32(num)
# define byteswap32(num) BSWAP_32(num)
# endif
# ifdef BSWAP_64
# define bswap64(num) BSWAP_64(num)
# define byteswap64(num) BSWAP_64(num)
# endif
# ifdef BE_16
# define conv16be(num) BE_16(num)
@ -119,15 +122,15 @@
# endif
#endif
#ifndef bswap16
# define bswap16(n) (uint16_t)( \
#ifndef byteswap16
# define byteswap16(n) (uint16_t)( \
(((n) & 0x00FFU) << 8) \
| (((n) & 0xFF00U) >> 8) \
)
#endif
#ifndef bswap32
# define bswap32(n) (uint32_t)( \
#ifndef byteswap32
# define byteswap32(n) (uint32_t)( \
(((n) & UINT32_C(0x000000FF)) << 24) \
| (((n) & UINT32_C(0x0000FF00)) << 8) \
| (((n) & UINT32_C(0x00FF0000)) >> 8) \
@ -135,8 +138,8 @@
)
#endif
#ifndef bswap64
# define bswap64(n) (uint64_t)( \
#ifndef byteswap64
# define byteswap64(n) (uint64_t)( \
(((n) & UINT64_C(0x00000000000000FF)) << 56) \
| (((n) & UINT64_C(0x000000000000FF00)) << 40) \
| (((n) & UINT64_C(0x0000000000FF0000)) << 24) \
@ -160,23 +163,23 @@
# define conv64be(num) ((uint64_t)(num))
# endif
# ifndef conv16le
# define conv16le(num) bswap16(num)
# define conv16le(num) byteswap16(num)
# endif
# ifndef conv32le
# define conv32le(num) bswap32(num)
# define conv32le(num) byteswap32(num)
# endif
# ifndef conv64le
# define conv64le(num) bswap64(num)
# define conv64le(num) byteswap64(num)
# endif
#else
# ifndef conv16be
# define conv16be(num) bswap16(num)
# define conv16be(num) byteswap16(num)
# endif
# ifndef conv32be
# define conv32be(num) bswap32(num)
# define conv32be(num) byteswap32(num)
# endif
# ifndef conv64be
# define conv64be(num) bswap64(num)
# define conv64be(num) byteswap64(num)
# endif
# ifndef conv16le
# define conv16le(num) ((uint16_t)(num))
@ -624,7 +627,7 @@ write64le(uint8_t *buf, uint64_t num)
// aligned but some compilers have language extensions to do that. With
// such language extensions the memcpy() method gives excellent results.
//
// What to do on a strict-align system when no known language extentensions
// What to do on a strict-align system when no known language extensions
// are available? Falling back to byte-by-byte access would be safe but ruin
// optimizations that have been made specifically with aligned access in mind.
// As a compromise, aligned reads will fall back to non-compliant type punning

4
src/liblzma/api/lzma.h
View file

@ -10,9 +10,9 @@
* headers) streams are supported. Multiple compression algorithms (filters)
* are supported. Currently LZMA2 is the primary filter.
*
* liblzma is part of XZ Utils <https://xz.tukaani.org/xz-utils/>. XZ Utils
* liblzma is part of XZ Utils <https://tukaani.org/xz/>. XZ Utils
* includes a gzip-like command line tool named xz and some other tools.
* XZ Utils is developed and maintained by Lasse Collin and Jia Tan.
* XZ Utils is developed and maintained by Lasse Collin.
*
* Major parts of liblzma are based on code written by Igor Pavlov,
* specifically the LZMA SDK <https://7-zip.org/sdk.html>.

2
src/liblzma/api/lzma/base.h
View file

@ -257,7 +257,7 @@ typedef enum {
*/
/*
* These eumerations may be used internally by liblzma
* These enumerations may be used internally by liblzma
* but they will never be returned to applications.
*/
LZMA_RET_INTERNAL1 = 101,

2
src/liblzma/api/lzma/container.h
View file

@ -792,7 +792,7 @@ extern LZMA_API(lzma_ret) lzma_stream_decoder_mt(
* as it doesn't support any decoder flags. It will return LZMA_STREAM_END
* after one .lzma stream.)
*
* \param strm Pointer to lzma_stream that is at least initialized
* \param strm Pointer to lzma_stream that is at least initialized
* with LZMA_STREAM_INIT.
* \param memlimit Memory usage limit as bytes. Use UINT64_MAX
* to effectively disable the limiter. liblzma

6
src/liblzma/api/lzma/index.h
View file

@ -233,7 +233,7 @@ typedef struct {
} block;
/**
* \private Internal struct.
* \private Internal data
*
* Internal data which is used to store the state of the iterator.
* The exact format may vary between liblzma versions, so don't
@ -673,7 +673,7 @@ extern LZMA_API(lzma_bool) lzma_index_iter_locate(
* function succeeds, the memory allocated for src
* is freed or moved to be part of dest, and all
* iterators pointing to src will become invalid.
* \param allocator lzma_allocator for custom allocator functions.
* \param allocator lzma_allocator for custom allocator functions.
* Set to NULL to use malloc() and free().
*
* \return Possible lzma_ret values:
@ -793,7 +793,7 @@ extern LZMA_API(lzma_ret) lzma_index_buffer_encode(const lzma_index *i,
* lzma_index is allowed to require. The value
* pointed by this pointer is modified if and only
* if LZMA_MEMLIMIT_ERROR is returned.
* \param allocator lzma_allocator for custom allocator functions.
* \param allocator lzma_allocator for custom allocator functions.
* Set to NULL to use malloc() and free().
* \param in Beginning of the input buffer
* \param in_pos The next byte will be read from in[*in_pos].

2
src/liblzma/api/lzma/version.h
View file

@ -22,7 +22,7 @@
#define LZMA_VERSION_MINOR 6
/** \brief Patch version number of the liblzma release. */
#define LZMA_VERSION_PATCH 0
#define LZMA_VERSION_PATCH 2
/**
* \brief Version stability marker

5
src/liblzma/api/lzma/vli.h
View file

@ -67,9 +67,8 @@ typedef uint64_t lzma_vli;
* This is useful to test that application has given acceptable values
* for example in the uncompressed_size and compressed_size variables.
*
* \return True if the integer is representable as VLI or if it
* indicates unknown value. False if the integer cannot be
* represented as VLI.
* \return True if the integer is representable as a VLI or if it
* indicates an unknown value. False otherwise.
*/
#define lzma_vli_is_valid(vli) \
((vli) <= LZMA_VLI_MAX || (vli) == LZMA_VLI_UNKNOWN)

25
src/liblzma/check/crc32_arm64.h
View file

@ -11,7 +11,6 @@
//
///////////////////////////////////////////////////////////////////////////////
#ifndef LZMA_CRC32_ARM64_H
#define LZMA_CRC32_ARM64_H
@ -21,6 +20,8 @@
# include <arm_acle.h>
#endif
// If both versions are going to be built, we need runtime detection
// to check if the instructions are supported.
#if defined(CRC32_GENERIC) && defined(CRC32_ARCH_OPTIMIZED)
# if defined(HAVE_GETAUXVAL) || defined(HAVE_ELF_AUX_INFO)
# include <sys/auxv.h>
@ -36,8 +37,7 @@
//
// NOTE: Build systems check for this too, keep them in sync with this.
#if (defined(__GNUC__) || defined(__clang__)) && !defined(__EDG__)
# define crc_attr_target \
__attribute__((__target__("+crc")))
# define crc_attr_target __attribute__((__target__("+crc")))
#else
# define crc_attr_target
#endif
@ -51,7 +51,7 @@ crc32_arch_optimized(const uint8_t *buf, size_t size, uint32_t crc)
// Align the input buffer because this was shown to be
// significantly faster than unaligned accesses.
const size_t align_amount = my_min(size, (8 - (uintptr_t)buf) & 7);
const size_t align_amount = my_min(size, (0U - (uintptr_t)buf) & 7);
for (const uint8_t *limit = buf + align_amount; buf < limit; ++buf)
crc = __crc32b(crc, *buf);
@ -62,7 +62,7 @@ crc32_arch_optimized(const uint8_t *buf, size_t size, uint32_t crc)
// ignoring the least significant three bits of size to ensure
// we do not process past the bounds of the buffer. This guarantees
// that limit is a multiple of 8 and is strictly less than size.
for (const uint8_t *limit = buf + (size & ~((size_t)7));
for (const uint8_t *limit = buf + (size & ~(size_t)7);
buf < limit; buf += 8)
crc = __crc32d(crc, aligned_read64le(buf));
@ -84,8 +84,10 @@ is_arch_extension_supported(void)
#elif defined(HAVE_ELF_AUX_INFO)
unsigned long feature_flags;
elf_aux_info(AT_HWCAP, &feature_flags, sizeof(feature_flags));
return feature_flags & HWCAP_CRC32 != 0;
if (elf_aux_info(AT_HWCAP, &feature_flags, sizeof(feature_flags)) != 0)
return false;
return (feature_flags & HWCAP_CRC32) != 0;
#elif defined(_WIN32)
return IsProcessorFeaturePresent(
@ -98,11 +100,12 @@ is_arch_extension_supported(void)
// The sysctlbyname() function requires a string identifier for the
// CPU feature it tests. The Apple documentation lists the string
// "hw.optional.armv8_crc32", which can be found here:
// (https://developer.apple.com/documentation/kernel/1387446-sysctlbyname/determining_instruction_set_characteristics#3915619)
int err = sysctlbyname("hw.optional.armv8_crc32", &has_crc32,
&size, NULL, 0);
// https://developer.apple.com/documentation/kernel/1387446-sysctlbyname/determining_instruction_set_characteristics#3915619
if (sysctlbyname("hw.optional.armv8_crc32", &has_crc32,
&size, NULL, 0) != 0)
return false;
return !err && has_crc32;
return has_crc32;
#else
// If a runtime detection method cannot be found, then this must

47
src/liblzma/check/crc32_fast.c
View file

@ -34,7 +34,7 @@ crc32_generic(const uint8_t *buf, size_t size, uint32_t crc)
crc = ~crc;
#ifdef WORDS_BIGENDIAN
crc = bswap32(crc);
crc = byteswap32(crc);
#endif
if (size > 8) {
@ -80,7 +80,7 @@ crc32_generic(const uint8_t *buf, size_t size, uint32_t crc)
crc = lzma_crc32_table[0][*buf++ ^ A(crc)] ^ S8(crc);
#ifdef WORDS_BIGENDIAN
crc = bswap32(crc);
crc = byteswap32(crc);
#endif
return ~crc;
@ -97,24 +97,14 @@ crc32_generic(const uint8_t *buf, size_t size, uint32_t crc)
// If both the generic and arch-optimized implementations are built, then
// the function to use is selected at runtime because the system running
// the binary might not have the arch-specific instruction set extension(s)
// available. The three dispatch methods in order of priority:
// available. The dispatch methods in order of priority:
//
// 1. Indirect function (ifunc). This method is slightly more efficient
// than the constructor method because it will change the entry in the
// Procedure Linkage Table (PLT) for the function either at load time or
// at the first call. This avoids having to call the function through a
// function pointer and will treat the function call like a regular call
// through the PLT. ifuncs are created by using
// __attribute__((__ifunc__("resolver"))) on a function which has no
// body. The "resolver" is the name of the function that chooses at
// runtime which implementation to use.
//
// 2. Constructor. This method uses __attribute__((__constructor__)) to
// 1. Constructor. This method uses __attribute__((__constructor__)) to
// set crc32_func at load time. This avoids extra computation (and any
// unlikely threading bugs) on the first call to lzma_crc32() to decide
// which implementation should be used.
//
// 3. First Call Resolution. On the very first call to lzma_crc32(), the
// 2. First Call Resolution. On the very first call to lzma_crc32(), the
// call will be directed to crc32_dispatch() instead. This will set the
// appropriate implementation function and will not be called again.
// This method does not use any kind of locking but is safe because if
@ -124,17 +114,7 @@ crc32_generic(const uint8_t *buf, size_t size, uint32_t crc)
typedef uint32_t (*crc32_func_type)(
const uint8_t *buf, size_t size, uint32_t crc);
// Clang 16.0.0 and older has a bug where it marks the ifunc resolver
// function as unused since it is static and never used outside of
// __attribute__((__ifunc__())).
#if defined(CRC_USE_IFUNC) && defined(__clang__)
# pragma GCC diagnostic push
# pragma GCC diagnostic ignored "-Wunused-function"
#endif
// This resolver is shared between all three dispatch methods. It serves as
// the ifunc resolver if ifunc is supported, otherwise it is called as a
// regular function by the constructor or first call resolution methods.
// This resolver is shared between all dispatch methods.
static crc32_func_type
crc32_resolve(void)
{
@ -142,11 +122,6 @@ crc32_resolve(void)
? &crc32_arch_optimized : &crc32_generic;
}
#if defined(CRC_USE_IFUNC) && defined(__clang__)
# pragma GCC diagnostic pop
#endif
#ifndef CRC_USE_IFUNC
#ifdef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR
// Constructor method.
@ -171,8 +146,7 @@ crc32_set_func(void)
static uint32_t
crc32_dispatch(const uint8_t *buf, size_t size, uint32_t crc)
{
// When __attribute__((__ifunc__(...))) and
// __attribute__((__constructor__)) isn't supported, set the
// When __attribute__((__constructor__)) isn't supported, set the
// function pointer without any locking. If multiple threads run
// the detection code in parallel, they will all end up setting
// the pointer to the same value. This avoids the use of
@ -184,14 +158,8 @@ crc32_dispatch(const uint8_t *buf, size_t size, uint32_t crc)
#endif
#endif
#endif
#ifdef CRC_USE_IFUNC
extern LZMA_API(uint32_t)
lzma_crc32(const uint8_t *buf, size_t size, uint32_t crc)
__attribute__((__ifunc__("crc32_resolve")));
#else
extern LZMA_API(uint32_t)
lzma_crc32(const uint8_t *buf, size_t size, uint32_t crc)
{
@ -234,4 +202,3 @@ lzma_crc32(const uint8_t *buf, size_t size, uint32_t crc)
return crc32_generic(buf, size, crc);
#endif
}
#endif

10
src/liblzma/check/crc32_table.c
View file

@ -17,18 +17,16 @@
#if defined(HAVE_USABLE_CLMUL) && ((defined(__x86_64__) && defined(__SSSE3__) \
&& defined(__SSE4_1__) && defined(__PCLMUL__)) \
|| (defined(__e2k__) && __iset__ >= 6))
# define X86_CLMUL_NO_TABLE 1
#endif
# define NO_CRC32_TABLE
#if defined(HAVE_ARM64_CRC32) \
#elif defined(HAVE_ARM64_CRC32) \
&& !defined(WORDS_BIGENDIAN) \
&& defined(__ARM_FEATURE_CRC32)
# define ARM64_CRC32_NO_TABLE 1
# define NO_CRC32_TABLE
#endif
#if !defined(HAVE_ENCODERS) && (defined(X86_CLMUL_NO_TABLE) \
|| defined(ARM64_CRC32_NO_TABLE_))
#if !defined(HAVE_ENCODERS) && defined(NO_CRC32_TABLE)
// No table needed. Use a typedef to avoid an empty translation unit.
typedef void lzma_crc32_dummy;

2
src/liblzma/check/crc32_tablegen.c
View file

@ -43,7 +43,7 @@ init_crc32_table(void)
#ifdef WORDS_BIGENDIAN
for (size_t s = 0; s < 8; ++s)
for (size_t b = 0; b < 256; ++b)
crc32_table[s][b] = bswap32(crc32_table[s][b]);
crc32_table[s][b] = byteswap32(crc32_table[s][b]);
#endif
return;

22
src/liblzma/check/crc64_fast.c
View file

@ -39,7 +39,7 @@ crc64_generic(const uint8_t *buf, size_t size, uint64_t crc)
crc = ~crc;
#ifdef WORDS_BIGENDIAN
crc = bswap64(crc);
crc = byteswap64(crc);
#endif
if (size > 4) {
@ -73,7 +73,7 @@ crc64_generic(const uint8_t *buf, size_t size, uint64_t crc)
crc = lzma_crc64_table[0][*buf++ ^ A1(crc)] ^ S8(crc);
#ifdef WORDS_BIGENDIAN
crc = bswap64(crc);
crc = byteswap64(crc);
#endif
return ~crc;
@ -93,11 +93,6 @@ crc64_generic(const uint8_t *buf, size_t size, uint64_t crc)
typedef uint64_t (*crc64_func_type)(
const uint8_t *buf, size_t size, uint64_t crc);
#if defined(CRC_USE_IFUNC) && defined(__clang__)
# pragma GCC diagnostic push
# pragma GCC diagnostic ignored "-Wunused-function"
#endif
static crc64_func_type
crc64_resolve(void)
{
@ -105,12 +100,6 @@ crc64_resolve(void)
? &crc64_arch_optimized : &crc64_generic;
}
#if defined(CRC_USE_IFUNC) && defined(__clang__)
# pragma GCC diagnostic pop
#endif
#ifndef CRC_USE_IFUNC
#ifdef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR
# define CRC64_SET_FUNC_ATTR __attribute__((__constructor__))
static crc64_func_type crc64_func;
@ -139,14 +128,8 @@ crc64_dispatch(const uint8_t *buf, size_t size, uint64_t crc)
}
#endif
#endif
#endif
#ifdef CRC_USE_IFUNC
extern LZMA_API(uint64_t)
lzma_crc64(const uint8_t *buf, size_t size, uint64_t crc)
__attribute__((__ifunc__("crc64_resolve")));
#else
extern LZMA_API(uint64_t)
lzma_crc64(const uint8_t *buf, size_t size, uint64_t crc)
{
@ -171,4 +154,3 @@ lzma_crc64(const uint8_t *buf, size_t size, uint64_t crc)
return crc64_generic(buf, size, crc);
#endif
}
#endif

4
src/liblzma/check/crc64_table.c
View file

@ -17,11 +17,11 @@
#if defined(HAVE_USABLE_CLMUL) && ((defined(__x86_64__) && defined(__SSSE3__) \
&& defined(__SSE4_1__) && defined(__PCLMUL__)) \
|| (defined(__e2k__) && __iset__ >= 6))
# define X86_CLMUL_NO_TABLE 1
# define NO_CRC64_TABLE
#endif
#ifdef X86_CLMUL_NO_TABLE
#ifdef NO_CRC64_TABLE
// No table needed. Use a typedef to avoid an empty translation unit.
typedef void lzma_crc64_dummy;

2
src/liblzma/check/crc64_tablegen.c
View file

@ -42,7 +42,7 @@ init_crc64_table(void)
#ifdef WORDS_BIGENDIAN
for (size_t s = 0; s < 4; ++s)
for (size_t b = 0; b < 256; ++b)
crc64_table[s][b] = bswap64(crc64_table[s][b]);
crc64_table[s][b] = byteswap64(crc64_table[s][b]);
#endif
return;

14
src/liblzma/check/crc_common.h
View file

@ -67,8 +67,6 @@
#undef CRC32_ARM64
#undef CRC64_ARM64_CLMUL
#undef CRC_USE_IFUNC
#undef CRC_USE_GENERIC_FOR_SMALL_INPUTS
// ARM64 CRC32 instruction is only useful for CRC32. Currently, only
@ -76,7 +74,7 @@
// endian machine.
//
// NOTE: Keep this and the next check in sync with the macro
// ARM64_CRC32_NO_TABLE in crc32_table.c
// NO_CRC32_TABLE in crc32_table.c
#if defined(HAVE_ARM64_CRC32) && !defined(WORDS_BIGENDIAN)
// Allow ARM64 CRC32 instruction without a runtime check if
// __ARM_FEATURE_CRC32 is defined. GCC and Clang only define this if the
@ -96,7 +94,8 @@
// generic version can be omitted. Note that this doesn't work with MSVC
// as I don't know how to detect the features here.
//
// NOTE: Keep this in sync with the CLMUL_NO_TABLE macro in crc32_table.c.
// NOTE: Keep this in sync with the NO_CRC32_TABLE macro in crc32_table.c
// and NO_CRC64_TABLE in crc64_table.c.
# if (defined(__SSSE3__) && defined(__SSE4_1__) && defined(__PCLMUL__)) \
|| (defined(__e2k__) && __iset__ >= 6)
# define CRC32_ARCH_OPTIMIZED 1
@ -109,9 +108,6 @@
# define CRC64_ARCH_OPTIMIZED 1
# define CRC_X86_CLMUL 1
# ifdef HAVE_FUNC_ATTRIBUTE_IFUNC
# define CRC_USE_IFUNC 1
# endif
/*
// The generic code is much faster with 1-8-byte inputs and
// has similar performance up to 16 bytes at least in
@ -121,9 +117,7 @@
// for bigger inputs. It saves a little in code size since
// the special cases for 0-16-byte inputs will be omitted
// from the CLMUL code.
# ifndef CRC_USE_IFUNC
# define CRC_USE_GENERIC_FOR_SMALL_INPUTS 1
# endif
# define CRC_USE_GENERIC_FOR_SMALL_INPUTS 1
*/
# endif
#endif

11
src/liblzma/check/crc_x86_clmul.h
View file

@ -385,15 +385,8 @@ crc64_arch_optimized(const uint8_t *buf, size_t size, uint64_t crc)
#endif // BUILDING_CRC64_CLMUL
// is_arch_extension_supported() must be inlined in this header file because
// the ifunc resolver function may not support calling a function in another
// translation unit. Depending on compiler-toolchain and flags, a call to
// a function defined in another translation unit could result in a
// reference to the PLT, which is unsafe to do in an ifunc resolver. The
// ifunc resolver runs very early when loading a shared library, so the PLT
// entries may not be setup at that time. Inlining this function duplicates
// the function body in crc32_resolve() and crc64_resolve(), but this is
// acceptable because the function results in very few instructions.
// Inlining this function duplicates the function body in crc32_resolve() and
// crc64_resolve(), but this is acceptable because this is a tiny function.
static inline bool
is_arch_extension_supported(void)
{

5
src/liblzma/common/easy_preset.h
View file

@ -9,6 +9,9 @@
//
///////////////////////////////////////////////////////////////////////////////
#ifndef LZMA_EASY_PRESET_H
#define LZMA_EASY_PRESET_H
#include "common.h"
@ -29,3 +32,5 @@ typedef struct {
/// Set *easy to the settings given by the preset. Returns true on error,
/// false on success.
extern bool lzma_easy_preset(lzma_options_easy *easy, uint32_t preset);
#endif

2
src/liblzma/common/filter_buffer_decoder.c
View file

@ -23,7 +23,7 @@ lzma_raw_buffer_decode(
|| out_pos == NULL || *out_pos > out_size)
return LZMA_PROG_ERROR;
// Initialize the decoer.
// Initialize the decoder.
lzma_next_coder next = LZMA_NEXT_CODER_INIT;
return_if_error(lzma_raw_decoder_init(&next, allocator, filters));

2
src/liblzma/common/filter_common.c
View file

@ -153,7 +153,7 @@ lzma_filters_copy(const lzma_filter *src, lzma_filter *real_dest,
return LZMA_PROG_ERROR;
// Use a temporary destination so that the real destination
// will never be modied if an error occurs.
// will never be modified if an error occurs.
lzma_filter dest[LZMA_FILTERS_MAX + 1];
lzma_ret ret;

15
src/liblzma/common/filter_decoder.c
View file

@ -150,6 +150,16 @@ decoder_find(lzma_vli id)
}
// lzma_filter_coder begins with the same members as lzma_filter_decoder.
// This function is a wrapper with a type that is compatible with the
// typedef of lzma_filter_find in filter_common.h.
static const lzma_filter_coder *
coder_find(lzma_vli id)
{
return (const lzma_filter_coder *)decoder_find(id);
}
extern LZMA_API(lzma_bool)
lzma_filter_decoder_is_supported(lzma_vli id)
{
@ -162,7 +172,7 @@ lzma_raw_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator,
const lzma_filter *options)
{
return lzma_raw_coder_init(next, allocator,
options, (lzma_filter_find)(&decoder_find), false);
options, &coder_find, false);
}
@ -181,8 +191,7 @@ lzma_raw_decoder(lzma_stream *strm, const lzma_filter *options)
extern LZMA_API(uint64_t)
lzma_raw_decoder_memusage(const lzma_filter *filters)
{
return lzma_raw_coder_memusage(
(lzma_filter_find)(&decoder_find), filters);
return lzma_raw_coder_memusage(&coder_find, filters);
}

17
src/liblzma/common/filter_encoder.c
View file

@ -193,6 +193,16 @@ encoder_find(lzma_vli id)
}
// lzma_filter_coder begins with the same members as lzma_filter_encoder.
// This function is a wrapper with a type that is compatible with the
// typedef of lzma_filter_find in filter_common.h.
static const lzma_filter_coder *
coder_find(lzma_vli id)
{
return (const lzma_filter_coder *)encoder_find(id);
}
extern LZMA_API(lzma_bool)
lzma_filter_encoder_is_supported(lzma_vli id)
{
@ -232,7 +242,7 @@ lzma_raw_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator,
const lzma_filter *filters)
{
return lzma_raw_coder_init(next, allocator,
filters, (lzma_filter_find)(&encoder_find), true);
filters, &coder_find, true);
}
@ -240,7 +250,7 @@ extern LZMA_API(lzma_ret)
lzma_raw_encoder(lzma_stream *strm, const lzma_filter *filters)
{
lzma_next_strm_init(lzma_raw_coder_init, strm, filters,
(lzma_filter_find)(&encoder_find), true);
&coder_find, true);
strm->internal->supported_actions[LZMA_RUN] = true;
strm->internal->supported_actions[LZMA_SYNC_FLUSH] = true;
@ -253,8 +263,7 @@ lzma_raw_encoder(lzma_stream *strm, const lzma_filter *filters)
extern LZMA_API(uint64_t)
lzma_raw_encoder_memusage(const lzma_filter *filters)
{
return lzma_raw_coder_memusage(
(lzma_filter_find)(&encoder_find), filters);
return lzma_raw_coder_memusage(&coder_find, filters);
}

11
src/liblzma/common/index_decoder.c
View file

@ -305,6 +305,12 @@ lzma_index_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator,
extern LZMA_API(lzma_ret)
lzma_index_decoder(lzma_stream *strm, lzma_index **i, uint64_t memlimit)
{
// If i isn't NULL, *i must always be initialized due to
// the wording in the API docs. This way it is initialized
// if we return LZMA_PROG_ERROR due to strm == NULL.
if (i != NULL)
*i = NULL;
lzma_next_strm_init(lzma_index_decoder_init, strm, i, memlimit);
strm->internal->supported_actions[LZMA_RUN] = true;
@ -319,6 +325,11 @@ lzma_index_buffer_decode(lzma_index **i, uint64_t *memlimit,
const lzma_allocator *allocator,
const uint8_t *in, size_t *in_pos, size_t in_size)
{
// If i isn't NULL, *i must always be initialized due to
// the wording in the API docs.
if (i != NULL)
*i = NULL;
// Sanity checks
if (i == NULL || memlimit == NULL
|| in == NULL || in_pos == NULL || *in_pos > in_size)

13
src/liblzma/common/memcmplen.h
View file

@ -67,6 +67,19 @@ lzma_memcmplen(const uint8_t *buf1, const uint8_t *buf2,
// This is only for x86-64 and ARM64 for now. This might be fine on
// other 64-bit processors too. On big endian one should use xor
// instead of subtraction and switch to __builtin_clzll().
//
// Reasons to use subtraction instead of xor:
//
// - On some x86-64 processors (Intel Sandy Bridge to Tiger Lake),
// sub+jz and sub+jnz can be fused but xor+jz or xor+jnz cannot.
// Thus using subtraction has potential to be a tiny amount faster
// since the code checks if the quotient is non-zero.
//
// - Some processors (Intel Pentium 4) used to have more ALU
// resources for add/sub instructions than and/or/xor.
//
// The processor info is based on Agner Fog's microarchitecture.pdf
// version 2023-05-26. https://www.agner.org/optimize/
#define LZMA_MEMCMPLEN_EXTRA 8
while (len < limit) {
const uint64_t x = read64ne(buf1 + len) - read64ne(buf2 + len);

5
src/liblzma/common/outqueue.h
View file

@ -9,6 +9,9 @@
//
///////////////////////////////////////////////////////////////////////////////
#ifndef LZMA_OUTQUEUE_H
#define LZMA_OUTQUEUE_H
#include "common.h"
@ -251,3 +254,5 @@ lzma_outq_outbuf_memusage(size_t buf_size)
assert(buf_size <= SIZE_MAX - sizeof(lzma_outbuf));
return sizeof(lzma_outbuf) + buf_size;
}
#endif

12
src/liblzma/common/string_conversion.c
View file

@ -217,12 +217,14 @@ typedef struct {
uint16_t offset;
union {
// NVHPC has problems with unions that contain pointers that
// are not the first members, so keep "map" at the top.
const name_value_map *map;
struct {
uint32_t min;
uint32_t max;
} range;
const name_value_map *map;
} u;
} option_map;
@ -1000,6 +1002,12 @@ extern LZMA_API(const char *)
lzma_str_to_filters(const char *str, int *error_pos, lzma_filter *filters,
uint32_t flags, const lzma_allocator *allocator)
{
// If error_pos isn't NULL, *error_pos must always be set.
// liblzma <= 5.4.6 and <= 5.6.1 have a bug and don't do this
// when str == NULL or filters == NULL or flags are unsupported.
if (error_pos != NULL)
*error_pos = 0;
if (str == NULL || filters == NULL)
return "Unexpected NULL pointer argument(s) "
"to lzma_str_to_filters()";

5
src/liblzma/delta/delta_decoder.c
View file

@ -25,6 +25,11 @@ decode_buffer(lzma_delta_coder *coder, uint8_t *buffer, size_t size)
}
// For an unknown reason NVIDIA HPC Compiler needs this pragma
// to produce working code.
#ifdef __NVCOMPILER
# pragma routine novector
#endif
static lzma_ret
delta_decode(void *coder_ptr, const lzma_allocator *allocator,
const uint8_t *restrict in, size_t *restrict in_pos,

2
src/liblzma/lz/lz_encoder.h
View file

@ -21,7 +21,7 @@
// changing this check.
#define IS_ENC_DICT_SIZE_VALID(size) \
((size) >= LZMA_DICT_SIZE_MIN \
&& (size) <= (UINT32_C(1) << 30) + (UINT32_C(1) << 29))
&& (size) <= (UINT32_C(1) << 30) + (UINT32_C(1) << 29))
/// A table of these is used by the LZ-based encoder to hold

1
src/liblzma/rangecoder/range_decoder.h
View file

@ -45,6 +45,7 @@
// and different processors. Overall 0x1F0 seems to be the best choice.
#ifndef LZMA_RANGE_DECODER_CONFIG
# if defined(__x86_64__) && !defined(__ILP32__) \
&& !defined(__NVCOMPILER) \
&& (defined(__GNUC__) || defined(__clang__))
# define LZMA_RANGE_DECODER_CONFIG 0x1F0
# else

6
src/liblzma/simple/riscv.c
View file

@ -116,7 +116,7 @@ AUIPC with rd != x0
Zfh, F, D, and Q:
* RV32I: LB, LH, LW, LBU, LHU, SB, SH, SW
* RV64I has also: LD, LWU, SD
* Zhf: FLH, FSH
* Zfh: FLH, FSH
* F: FLW, FSW
* D: FLD, FSD
* Q: FLQ, FSQ
@ -320,11 +320,11 @@ AUIPC with rd == x0
// The left-hand side takes care of (1) and (2).
// (a) The lowest 7 bits are already known to be AUIPC so subtracting 0x17
// makes those bits zeros.
// (b) If AUIPC rd equals x2, subtracting 0x10 makes bits [11:7] zeros.
// (b) If AUIPC rd equals x2, subtracting 0x100 makes bits [11:7] zeros.
// If rd doesn't equal x2, then there will be at least one non-zero bit
// and the next step (c) is irrelevant.
// (c) If the lowest two opcode bits of the packed inst2 are set in [13:12],
// then subtracting 0x300 will make those bits zeros. Otherwise there
// then subtracting 0x3000 will make those bits zeros. Otherwise there
// will be at least one non-zero bit.
//
// The shift by 18 removes the high bits from the final '>=' comparison and

8
src/liblzma/simple/simple_coder.h
View file

@ -61,12 +61,12 @@ extern lzma_ret lzma_simple_armthumb_decoder_init(lzma_next_coder *next,
extern lzma_ret lzma_simple_arm64_encoder_init(lzma_next_coder *next,
const lzma_allocator *allocator,
const lzma_filter_info *filters);
const lzma_allocator *allocator,
const lzma_filter_info *filters);
extern lzma_ret lzma_simple_arm64_decoder_init(lzma_next_coder *next,
const lzma_allocator *allocator,
const lzma_filter_info *filters);
const lzma_allocator *allocator,
const lzma_filter_info *filters);
extern lzma_ret lzma_simple_sparc_encoder_init(lzma_next_coder *next,

11
src/liblzma/simple/x86.c
View file

@ -26,11 +26,7 @@ static size_t
x86_code(void *simple_ptr, uint32_t now_pos, bool is_encoder,
uint8_t *buffer, size_t size)
{
static const bool MASK_TO_ALLOWED_STATUS[8]
= { true, true, true, false, true, false, false, false };
static const uint32_t MASK_TO_BIT_NUMBER[8]
= { 0, 1, 2, 2, 3, 3, 3, 3 };
static const uint32_t MASK_TO_BIT_NUMBER[5] = { 0, 1, 2, 2, 3 };
lzma_simple_x86 *simple = simple_ptr;
uint32_t prev_mask = simple->prev_mask;
@ -67,9 +63,8 @@ x86_code(void *simple_ptr, uint32_t now_pos, bool is_encoder,
b = buffer[buffer_pos + 4];
if (Test86MSByte(b)
&& MASK_TO_ALLOWED_STATUS[(prev_mask >> 1) & 0x7]
&& (prev_mask >> 1) < 0x10) {
if (Test86MSByte(b) && (prev_mask >> 1) <= 4
&& (prev_mask >> 1) != 3) {
uint32_t src = ((uint32_t)(b) << 24)
| ((uint32_t)(buffer[buffer_pos + 3]) << 16)

35
src/xz/args.c
View file

@ -92,6 +92,12 @@ parse_block_list(const char *str_const)
free(opt_block_list);
opt_block_list = xmalloc((count + 1) * sizeof(block_list_entry));
// Clear the bitmask of filter chains in use.
block_list_chain_mask = 0;
// Reset the largest Block size found in --block-list.
block_list_largest = 0;
for (size_t i = 0; i < count; ++i) {
// Locate the next comma and replace it with \0.
char *p = strchr(str, ',');
@ -99,7 +105,7 @@ parse_block_list(const char *str_const)
*p = '\0';
// Use the default filter chain unless overridden.
opt_block_list[i].filters_index = 0;
opt_block_list[i].chain_num = 0;
// To specify a filter chain, the block list entry may be
// prepended with "[filter-chain-number]:". The size is
@ -126,10 +132,13 @@ parse_block_list(const char *str_const)
"filter chain number '%c:'"),
str[0]);
int filter_num = str[0] - '0';
opt_block_list[i].filters_index =
(uint32_t)filter_num;
const unsigned chain_num = (unsigned)(str[0] - '0');
opt_block_list[i].chain_num = chain_num;
block_list_chain_mask |= 1U << chain_num;
str += 2;
} else {
// This Block uses the default filter chain.
block_list_chain_mask |= 1U << 0;
}
if (str[0] == '\0') {
@ -153,9 +162,23 @@ parse_block_list(const char *str_const)
opt_block_list[i].size = UINT64_MAX;
}
// Remember the largest Block size in the list.
//
// NOTE: Do this after handling the special value 0
// because when 0 is used, we don't want to reduce
// the Block size of the multithreaded encoder.
if (block_list_largest < opt_block_list[i].size)
block_list_largest = opt_block_list[i].size;
}
str = p + 1;
// Be standards compliant: p + 1 is undefined behavior
// if p == NULL. That occurs on the last iteration of
// the loop when we won't care about the value of str
// anymore anyway. That is, this is done conditionally
// solely for standard conformance reasons.
if (p != NULL)
str = p + 1;
}
// Terminate the array.
@ -238,7 +261,7 @@ parse_real(args_info *args, int argc, char **argv)
{ "check", required_argument, NULL, 'C' },
{ "ignore-check", no_argument, NULL, OPT_IGNORE_CHECK },
{ "block-size", required_argument, NULL, OPT_BLOCK_SIZE },
{ "block-list", required_argument, NULL, OPT_BLOCK_LIST },
{ "block-list", required_argument, NULL, OPT_BLOCK_LIST },
{ "memlimit-compress", required_argument, NULL, OPT_MEM_COMPRESS },
{ "memlimit-decompress", required_argument, NULL, OPT_MEM_DECOMPRESS },
{ "memlimit-mt-decompress", required_argument, NULL, OPT_MEM_MT_DECOMPRESS },

495
src/xz/coder.c
View file

@ -11,6 +11,7 @@
///////////////////////////////////////////////////////////////////////////////
#include "private.h"
#include "tuklib_integer.h"
/// Return value type for coder_init().
@ -27,6 +28,8 @@ bool opt_auto_adjust = true;
bool opt_single_stream = false;
uint64_t opt_block_size = 0;
block_list_entry *opt_block_list = NULL;
uint64_t block_list_largest;
uint32_t block_list_chain_mask;
/// Stream used to communicate with liblzma
static lzma_stream strm = LZMA_STREAM_INIT;
@ -35,25 +38,19 @@ static lzma_stream strm = LZMA_STREAM_INIT;
/// and 9 other filter chains can be specified with --filtersX.
#define NUM_FILTER_CHAIN_MAX 10
/// The default filter chain is in filters[0]. It is used for encoding
/// The default filter chain is in chains[0]. It is used for encoding
/// in all supported formats and also for decdoing raw streams. The other
/// filter chains are set by --filtersX to support changing filters with
/// the --block-list option.
static lzma_filter filters[NUM_FILTER_CHAIN_MAX][LZMA_FILTERS_MAX + 1];
static lzma_filter chains[NUM_FILTER_CHAIN_MAX][LZMA_FILTERS_MAX + 1];
/// Bit mask representing the filters that are actually used when encoding
/// in the xz format. This is needed since a filter chain could be
/// specified in --filtersX (or the default filter chain), but never used
/// in --block-list. The default filter chain is always assumed to be used,
/// unless --block-list is specified and does not have a block using the
/// default filter chain.
static uint32_t filters_used_mask = 1;
#ifdef HAVE_ENCODERS
/// Track the memory usage for all filter chains (default or --filtersX).
/// The memory usage may need to be scaled down depending on the memory limit.
static uint64_t filter_memusages[ARRAY_SIZE(filters)];
#endif
/// Bitmask indicating which filter chains are actually used when encoding
/// in the .xz format. This is needed since the filter chains specified using
/// --filtersX (or the default filter chain) might in reality be unneeded
/// if they are never used in --block-list. When --block-list isn't
/// specified, only the default filter chain is used, thus the initial
/// value of this variable is 1U << 0 (the number of the default chain is 0).
static uint32_t chains_used_mask = 1U << 0;
/// Input and output buffers
static io_buf in_buf;
@ -105,7 +102,7 @@ forget_filter_chain(void)
// Setting a preset or using --filters makes us forget
// the earlier custom filter chain (if any).
if (filters_count > 0) {
lzma_filters_free(filters[0], NULL);
lzma_filters_free(chains[0], NULL);
filters_count = 0;
}
@ -142,11 +139,12 @@ coder_add_filter(lzma_vli id, void *options)
if (string_to_filter_used)
forget_filter_chain();
filters[0][filters_count].id = id;
filters[0][filters_count].options = options;
chains[0][filters_count].id = id;
chains[0][filters_count].options = options;
// Terminate the filter chain with LZMA_VLI_UNKNOWN to simplify
// implementation of forget_filter_chain().
filters[0][++filters_count].id = LZMA_VLI_UNKNOWN;
chains[0][++filters_count].id = LZMA_VLI_UNKNOWN;
// Setting a custom filter chain makes us forget the preset options.
// This makes a difference if one specifies e.g. "xz -9 --lzma2 -e"
@ -163,7 +161,7 @@ str_to_filters(const char *str, uint32_t index, uint32_t flags)
{
int error_pos;
const char *err = lzma_str_to_filters(str, &error_pos,
filters[index], flags, NULL);
chains[index], flags, NULL);
if (err != NULL) {
char filter_num[2] = "";
@ -199,7 +197,7 @@ coder_add_filters_from_str(const char *filter_str)
// Set the filters_count to be the number of filters converted from
// the string.
for (filters_count = 0; filters[0][filters_count].id
for (filters_count = 0; chains[0][filters_count].id
!= LZMA_VLI_UNKNOWN;
++filters_count) ;
@ -212,12 +210,12 @@ extern void
coder_add_block_filters(const char *str, size_t slot)
{
// Free old filters first, if they were previously allocated.
if (filters_used_mask & (1U << slot))
lzma_filters_free(filters[slot], NULL);
if (chains_used_mask & (1U << slot))
lzma_filters_free(chains[slot], NULL);
str_to_filters(str, slot, 0);
filters_used_mask |= 1U << slot;
chains_used_mask |= 1U << slot;
}
@ -233,29 +231,26 @@ memlimit_too_small(uint64_t memory_usage)
#ifdef HAVE_ENCODERS
// For a given opt_block_list index, validate that the filter has been
// set. If it has not been set, we must exit with error to avoid using
// an uninitialized filter chain.
static void
validate_block_list_filter(const uint32_t filter_num)
{
if (!(filters_used_mask & (1U << filter_num)))
message_fatal(_("filter chain %u used by --block-list but "
"not specified with --filters%u="),
(unsigned)filter_num, (unsigned)filter_num);
}
// Sets the memory usage for each filter chain. It will return the maximum
// memory usage of all of the filter chains.
/// \brief Calculate the memory usage of each filter chain.
///
/// \param chains_memusages If non-NULL, the memusage of the encoder
/// or decoder for each chain is stored in
/// this array.
/// \param mt If non-NULL, calculate memory usage of
/// multithreaded encoder.
/// \param encode Whether to calculate encoder or decoder
/// memory usage. This must be true if
/// mt != NULL.
///
/// \return Return the highest memory usage of all of the filter chains.
static uint64_t
filters_memusage_max(const lzma_mt *mt, bool encode)
get_chains_memusage(uint64_t *chains_memusages, const lzma_mt *mt, bool encode)
{
uint64_t max_memusage = 0;
#ifdef MYTHREAD_ENABLED
// Copy multithreaded options to a temporary struct since the
// filters member needs to be changed
// Copy multithreading options to a temporary struct since the
// "filters" member needs to be changed.
lzma_mt mt_local;
if (mt != NULL)
mt_local = *mt;
@ -263,40 +258,39 @@ filters_memusage_max(const lzma_mt *mt, bool encode)
(void)mt;
#endif
for (uint32_t i = 0; i < ARRAY_SIZE(filters); i++) {
if (!(filters_used_mask & (1U << i)))
for (uint32_t i = 0; i < ARRAY_SIZE(chains); i++) {
if (!(chains_used_mask & (1U << i)))
continue;
uint64_t memusage = UINT64_MAX;
#ifdef MYTHREAD_ENABLED
if (mt != NULL) {
mt_local.filters = filters[i];
assert(encode);
mt_local.filters = chains[i];
memusage = lzma_stream_encoder_mt_memusage(&mt_local);
filter_memusages[i] = memusage;
}
else
} else
#endif
if (encode) {
memusage = lzma_raw_encoder_memusage(filters[i]);
filter_memusages[i] = memusage;
memusage = lzma_raw_encoder_memusage(chains[i]);
}
#ifdef HAVE_DECODERS
else {
memusage = lzma_raw_decoder_memusage(filters[i]);
memusage = lzma_raw_decoder_memusage(chains[i]);
}
#endif
if (chains_memusages != NULL)
chains_memusages[i] = memusage;
if (memusage > max_memusage)
max_memusage = memusage;
}
return max_memusage;
}
#endif
extern void
coder_set_compression_settings(void)
{
@ -305,48 +299,6 @@ coder_set_compression_settings(void)
assert(opt_format != FORMAT_LZIP);
#endif
#ifdef HAVE_ENCODERS
# ifdef MYTHREAD_ENABLED
// Represents the largest Block size specified with --block-list. This
// is needed to help reduce the Block size in the multithreaded encoder
// so memory is not wasted.
uint64_t max_block_list_size = 0;
# endif
if (opt_block_list != NULL) {
// This mask tracks the filters actually referenced in
// --block-list. It is used to help remove bits from
// filters_used_mask when a filter chain was specified
// but never actually used.
uint32_t filters_ref_mask = 0;
for (uint32_t i = 0; opt_block_list[i].size != 0; i++) {
validate_block_list_filter(
opt_block_list[i].filters_index);
// Mark the current filter as referenced.
filters_ref_mask |= 1U <<
opt_block_list[i].filters_index;
# ifdef MYTHREAD_ENABLED
if (opt_block_list[i].size > max_block_list_size)
max_block_list_size = opt_block_list[i].size;
# endif
}
assert(filters_ref_mask != 0);
// Note: The filters that were initialized but not used do
// not free their options and do not have the filter
// IDs set to LZMA_VLI_UNKNOWN. Filter chains are not
// freed outside of debug mode and the default filter
// chain is never freed.
filters_used_mask = filters_ref_mask;
} else {
// Reset filters used mask in case --block-list is not
// used, but --filtersX is used.
filters_used_mask = 1;
}
#endif
// The default check type is CRC64, but fallback to CRC32
// if CRC64 isn't supported by the copy of liblzma we are
// using. CRC32 is always supported.
@ -356,14 +308,56 @@ coder_set_compression_settings(void)
check = LZMA_CHECK_CRC32;
}
#ifdef HAVE_ENCODERS
if (opt_block_list != NULL) {
// args.c ensures these.
assert(opt_mode == MODE_COMPRESS);
assert(opt_format == FORMAT_XZ);
// Find out if block_list_chain_mask has a bit set that
// isn't set in chains_used_mask.
const uint32_t missing_chains_mask
= (block_list_chain_mask ^ chains_used_mask)
& block_list_chain_mask;
// If a filter chain was specified in --block-list but no
// matching --filtersX option was used, exit with an error.
if (missing_chains_mask != 0) {
// Get the number of the first missing filter chain
// and show it in the error message.
const unsigned first_missing
= (unsigned)ctz32(missing_chains_mask);
message_fatal(_("filter chain %u used by "
"--block-list but not specified "
"with --filters%u="),
first_missing, first_missing);
}
// Omit the unused filter chains from mask of used chains.
//
// (FIXME? When built with debugging, coder_free() will free()
// the filter chains (except the default chain) which makes
// Valgrind show fewer reachable allocations. But coder_free()
// uses this mask to determine which chains to free. Thus it
// won't free the ones that are cleared here from the mask.
// In practice this doesn't matter.)
chains_used_mask &= block_list_chain_mask;
} else {
// Reset filters used mask in case --block-list is not
// used, but --filtersX is used.
chains_used_mask = 1U << 0;
}
#endif
// Options for LZMA1 or LZMA2 in case we are using a preset.
static lzma_options_lzma opt_lzma;
// The first filter in the filters[] array is for the default
// The first filter in the chains[] array is for the default
// filter chain.
lzma_filter *default_filters = filters[0];
lzma_filter *default_filters = chains[0];
if (filters_count == 0 && filters_used_mask & 1) {
if (filters_count == 0 && chains_used_mask & 1) {
// We are using a preset. This is not a good idea in raw mode
// except when playing around with things. Different versions
// of this software may use different options in presets, and
@ -403,14 +397,16 @@ coder_set_compression_settings(void)
"the LZMA1 filter"));
// If we are using the .xz format, make sure that there is no LZMA1
// filter to prevent LZMA_PROG_ERROR.
if (opt_format == FORMAT_XZ && filters_used_mask & 1)
// filter to prevent LZMA_PROG_ERROR. With the chains from --filtersX
// we have already ensured this by calling lzma_str_to_filters()
// without setting the flags that would allow non-.xz filters.
if (opt_format == FORMAT_XZ && chains_used_mask & 1)
for (size_t i = 0; i < filters_count; ++i)
if (default_filters[i].id == LZMA_FILTER_LZMA1)
message_fatal(_("LZMA1 cannot be used "
"with the .xz format"));
if (filters_used_mask & 1) {
if (chains_used_mask & 1) {
// Print the selected default filter chain.
message_filters_show(V_DEBUG, default_filters);
}
@ -419,11 +415,11 @@ coder_set_compression_settings(void)
// from the filter chain. Currently the threaded encoder doesn't
// support LZMA_SYNC_FLUSH so single-threaded mode must be used.
if (opt_mode == MODE_COMPRESS && opt_flush_timeout != 0) {
for (uint32_t i = 0; i < ARRAY_SIZE(filters); ++i) {
if (!(filters_used_mask & (1U << i)))
for (unsigned i = 0; i < ARRAY_SIZE(chains); ++i) {
if (!(chains_used_mask & (1U << i)))
continue;
const lzma_filter *fc = filters[i];
const lzma_filter *fc = chains[i];
for (size_t j = 0; fc[j].id != LZMA_VLI_UNKNOWN; j++) {
switch (fc[j].id) {
case LZMA_FILTER_LZMA2:
@ -434,7 +430,7 @@ coder_set_compression_settings(void)
message_fatal(_("Filter chain %u is "
"incompatible with "
"--flush-timeout"),
(unsigned)i);
i);
}
}
}
@ -446,17 +442,22 @@ coder_set_compression_settings(void)
}
}
// Get the memory usage and memory limit. The memory usage is the
// maximum of the default filters[] and any filters specified by
// --filtersX.
// Note that if --format=raw was used, we can be decompressing and
// do not need to account for any filter chains created
// with --filtersX.
// Get memory limit and the memory usage of the used filter chains.
// Note that if --format=raw was used, we can be decompressing
// using the default filter chain.
//
// If multithreaded .xz compression is done, the memory limit
// will be replaced.
uint64_t memory_limit = hardware_memlimit_get(opt_mode);
uint64_t memory_usage = UINT64_MAX;
#ifdef HAVE_ENCODERS
// Memory usage for each encoder filter chain (default
// or --filtersX). The encoder options may need to be
// scaled down depending on the memory usage limit.
uint64_t encoder_memusages[ARRAY_SIZE(chains)];
#endif
if (opt_mode == MODE_COMPRESS) {
#ifdef HAVE_ENCODERS
# ifdef MYTHREAD_ENABLED
@ -465,16 +466,17 @@ coder_set_compression_settings(void)
mt_options.threads = hardware_threads_get();
uint64_t block_size = opt_block_size;
// If opt_block_size is not set, find the maximum
// recommended Block size based on the filter chains
if (block_size == 0) {
for (uint32_t i = 0; i < ARRAY_SIZE(filters);
for (unsigned i = 0; i < ARRAY_SIZE(chains);
i++) {
if (!(filters_used_mask & (1U << i)))
if (!(chains_used_mask & (1U << i)))
continue;
uint64_t size = lzma_mt_block_size(
filters[i]);
chains[i]);
// If this returns an error, then one
// of the filter chains in use is
@ -483,33 +485,28 @@ coder_set_compression_settings(void)
if (size == UINT64_MAX)
message_fatal(_("Unsupported "
"options in filter "
"chain %u"),
(unsigned)i);
"chain %u"), i);
if (size > block_size)
block_size = size;
}
// If the largest block size specified
// with --block-list is less than the
// recommended Block size, then it is a waste
// of RAM to use a larger Block size. It may
// even allow more threads to be used in some
// situations. If the special 0 Block size is
// used (encode all remaining data in 1 Block)
// then max_block_list_size will be set to
// UINT64_MAX, so the recommended Block size
// will always be used in this case.
if (max_block_list_size > 0
&& max_block_list_size
< block_size)
block_size = max_block_list_size;
// If --block-list was used and our current
// Block size exceeds the largest size
// in --block-list, reduce the Block size of
// the multithreaded encoder. The extra size
// would only be a waste of RAM. With a
// smaller Block size we might even be able
// to use more threads in some cases.
if (block_list_largest > 0 && block_size
> block_list_largest)
block_size = block_list_largest;
}
mt_options.block_size = block_size;
mt_options.check = check;
memory_usage = filters_memusage_max(
memory_usage = get_chains_memusage(encoder_memusages,
&mt_options, true);
if (memory_usage != UINT64_MAX)
message(V_DEBUG, _("Using up to %" PRIu32
@ -518,7 +515,8 @@ coder_set_compression_settings(void)
} else
# endif
{
memory_usage = filters_memusage_max(NULL, true);
memory_usage = get_chains_memusage(encoder_memusages,
NULL, true);
}
#endif
} else {
@ -533,21 +531,13 @@ coder_set_compression_settings(void)
// Print memory usage info before possible dictionary
// size auto-adjusting.
//
// NOTE: If only encoder support was built, we cannot show the
// NOTE: If only encoder support was built, we cannot show
// what the decoder memory usage will be.
message_mem_needed(V_DEBUG, memory_usage);
#ifdef HAVE_DECODERS
if (opt_mode == MODE_COMPRESS) {
#ifdef HAVE_ENCODERS
const uint64_t decmem =
filters_memusage_max(NULL, false);
#else
// If encoders are not enabled, then --block-list is never
// usable, so the other filter chains 1-9 can never be used.
// So there is no need to find the maximum decoder memory
// required in this case.
const uint64_t decmem = lzma_raw_decoder_memusage(filters[0]);
#endif
#if defined(HAVE_ENCODERS) && defined(HAVE_DECODERS)
if (opt_mode == MODE_COMPRESS && message_verbosity_get() >= V_DEBUG) {
const uint64_t decmem = get_chains_memusage(NULL, NULL, false);
if (decmem != UINT64_MAX)
message(V_DEBUG, _("Decompression will need "
"%s MiB of memory."), uint64_to_str(
@ -574,14 +564,21 @@ coder_set_compression_settings(void)
// Reduce the number of threads by one and check
// the memory usage.
--mt_options.threads;
memory_usage = filters_memusage_max(
memory_usage = get_chains_memusage(encoder_memusages,
&mt_options, true);
if (memory_usage == UINT64_MAX)
message_bug();
if (memory_usage <= memory_limit) {
// The memory usage is now low enough.
message(V_WARNING, _("Reduced the number of "
//
// Since 5.6.1: This is only shown at
// V_DEBUG instead of V_WARNING because
// changing the number of threads doesn't
// affect the output. On some systems this
// message would be too common now that
// multithreaded compression is the default.
message(V_DEBUG, _("Reduced the number of "
"threads from %s to %s to not exceed "
"the memory usage limit of %s MiB"),
uint64_to_str(
@ -600,8 +597,11 @@ coder_set_compression_settings(void)
// way -T0 won't use insane amount of memory but at the same
// time the soft limit will never make xz fail and never make
// xz change settings that would affect the compressed output.
//
// Since 5.6.1: Like above, this is now shown at V_DEBUG
// instead of V_WARNING.
if (hardware_memlimit_mtenc_is_default()) {
message(V_WARNING, _("Reduced the number of threads "
message(V_DEBUG, _("Reduced the number of threads "
"from %s to one. The automatic memory usage "
"limit of %s MiB is still being exceeded. "
"%s MiB of memory is required. "
@ -627,7 +627,8 @@ coder_set_compression_settings(void)
// the multithreaded mode but the output
// is also different.
hardware_threads_set(1);
memory_usage = filters_memusage_max(NULL, true);
memory_usage = get_chains_memusage(encoder_memusages,
NULL, true);
message(V_WARNING, _("Switching to single-threaded mode "
"to not exceed the memory usage limit of %s MiB"),
uint64_to_str(round_up_to_mib(memory_limit), 0));
@ -642,137 +643,84 @@ coder_set_compression_settings(void)
if (!opt_auto_adjust)
memlimit_too_small(memory_usage);
// Decrease the dictionary size until we meet the memory usage limit.
// The struct is used to track data needed to correctly reduce the
// memory usage and report which filters were adjusted.
typedef struct {
// Pointer to the filter chain that needs to be reduced.
// NULL indicates that this filter chain was either never
// set or was never above the memory limit.
lzma_filter *filters;
// Original dictionary sizes are used to show how each
// filter's dictionary was reduced.
uint64_t orig_dict_size;
// Index of the LZMA filter in the filters member. We only
// adjust this filter's memusage because we don't know how
// to reduce the memory usage of the other filters.
uint32_t lzma_idx;
// Indicates if the filter's dictionary size needs to be
// reduced to fit under the memory limit (true) or if the
// filter chain is unused or is already under the memory
// limit (false).
bool reduce_dict_size;
} memusage_reduction_data;
memusage_reduction_data memusage_reduction[ARRAY_SIZE(filters)];
// Counter represents how many filter chains are above the memory
// limit.
size_t count = 0;
for (uint32_t i = 0; i < ARRAY_SIZE(filters); i++) {
// The short var name "r" will reduce the number of lines
// of code needed since less lines will stretch past 80
// characters.
memusage_reduction_data *r = &memusage_reduction[i];
r->filters = NULL;
r->reduce_dict_size = false;
if (!(filters_used_mask & (1U << i)))
// Adjust each filter chain that is exceeding the memory usage limit.
for (unsigned i = 0; i < ARRAY_SIZE(chains); i++) {
// Skip unused chains.
if (!(chains_used_mask & (1U << i)))
continue;
for (uint32_t j = 0; filters[i][j].id != LZMA_VLI_UNKNOWN;
j++)
if ((filters[i][j].id == LZMA_FILTER_LZMA2
|| filters[i][j].id
== LZMA_FILTER_LZMA1)
&& filter_memusages[i]
> memory_limit) {
count++;
r->filters = filters[i];
r->lzma_idx = j;
r->reduce_dict_size = true;
// Skip chains that already meet the memory usage limit.
if (encoder_memusages[i] <= memory_limit)
continue;
lzma_options_lzma *opt = r->filters
[r->lzma_idx].options;
r->orig_dict_size = opt->dict_size;
opt->dict_size &= ~((UINT32_C(1) << 20) - 1);
}
}
// Look for the last filter if it is LZMA2 or LZMA1, so we
// can make it use less RAM. We cannot adjust other filters.
unsigned j = 0;
while (chains[i][j].id != LZMA_FILTER_LZMA2
&& chains[i][j].id != LZMA_FILTER_LZMA1) {
// NOTE: This displays the too high limit of this
// particular filter chain. If multiple chains are
// specified and another one would need more then
// this message could be confusing. As long as LZMA2
// is the only memory hungry filter in .xz this
// doesn't matter at all in practice.
//
// FIXME? However, it's sort of odd still if we had
// switched from multithreaded mode to single-threaded
// mode because single-threaded produces different
// output. So the messages could perhaps be clearer.
// Another case of this is a few lines below.
if (chains[i][j].id == LZMA_VLI_UNKNOWN)
memlimit_too_small(encoder_memusages[i]);
// Loop until all filters use <= memory_limit, or exit.
while (count > 0) {
for (uint32_t i = 0; i < ARRAY_SIZE(memusage_reduction); i++) {
memusage_reduction_data *r = &memusage_reduction[i];
++j;
}
if (!r->reduce_dict_size)
continue;
lzma_options_lzma *opt =
r->filters[r->lzma_idx].options;
// Decrease the dictionary size until we meet the memory
// usage limit. First round down to full mebibytes.
lzma_options_lzma *opt = chains[i][j].options;
const uint32_t orig_dict_size = opt->dict_size;
opt->dict_size &= ~((UINT32_C(1) << 20) - 1);
while (true) {
// If it is below 1 MiB, auto-adjusting failed.
// We could be more sophisticated and scale it
// down even more, but nobody has complained so far.
//
// FIXME? See the FIXME a few lines above.
if (opt->dict_size < (UINT32_C(1) << 20))
memlimit_too_small(memory_usage);
memlimit_too_small(encoder_memusages[i]);
uint64_t filt_mem_usage =
lzma_raw_encoder_memusage(r->filters);
if (filt_mem_usage == UINT64_MAX)
encoder_memusages[i]
= lzma_raw_encoder_memusage(chains[i]);
if (encoder_memusages[i] == UINT64_MAX)
message_bug();
if (filt_mem_usage < memory_limit) {
r->reduce_dict_size = false;
count--;
}
else {
opt->dict_size -= UINT32_C(1) << 20;
}
// Accept it if it is low enough.
if (encoder_memusages[i] <= memory_limit)
break;
// Otherwise adjust it 1 MiB down and try again.
opt->dict_size -= UINT32_C(1) << 20;
}
}
// Tell the user that we decreased the dictionary size for
// each filter that was adjusted.
for (uint32_t i = 0; i < ARRAY_SIZE(memusage_reduction); i++) {
memusage_reduction_data *r = &memusage_reduction[i];
// If the filters were never set, then the memory usage
// was never adjusted.
if (r->filters == NULL)
continue;
lzma_filter *filter_lzma = &(r->filters[r->lzma_idx]);
lzma_options_lzma *opt = filter_lzma->options;
// The first index is the default filter chain. The message
// should be slightly different if the default filter chain
// or if --filtersX was adjusted.
// Tell the user that we decreased the dictionary size.
// The message is slightly different between the default
// filter chain (0) or and chains from --filtersX.
const char lzma_num = chains[i][j].id == LZMA_FILTER_LZMA2
? '2' : '1';
const char *from_size = uint64_to_str(orig_dict_size >> 20, 0);
const char *to_size = uint64_to_str(opt->dict_size >> 20, 1);
const char *limit_size = uint64_to_str(round_up_to_mib(
memory_limit), 2);
if (i == 0)
message(V_WARNING, _("Adjusted LZMA%c dictionary "
"size from %s MiB to %s MiB to not exceed the "
"memory usage limit of %s MiB"),
filter_lzma->id == LZMA_FILTER_LZMA2
? '2' : '1',
uint64_to_str(r->orig_dict_size >> 20, 0),
uint64_to_str(opt->dict_size >> 20, 1),
uint64_to_str(round_up_to_mib(
memory_limit), 2));
lzma_num, from_size, to_size, limit_size);
else
message(V_WARNING, _("Adjusted LZMA%c dictionary size "
"for --filters%u from %s MiB to %s MiB to not "
"exceed the memory usage limit of %s MiB"),
filter_lzma->id == LZMA_FILTER_LZMA2
? '2' : '1',
(unsigned)i,
uint64_to_str(r->orig_dict_size >> 20, 0),
uint64_to_str(opt->dict_size >> 20, 1),
uint64_to_str(round_up_to_mib(
memory_limit), 2));
lzma_num, i, from_size, to_size, limit_size);
}
#endif
@ -870,11 +818,11 @@ coder_init(file_pair *pair)
allow_trailing_input = false;
// Set the first filter chain. If the --block-list option is not
// used then use the default filter chain (filters[0]).
// used then use the default filter chain (chains[0]).
// Otherwise, use first filter chain from the block list.
lzma_filter *active_filters = opt_block_list == NULL
? filters[0]
: filters[opt_block_list[0].filters_index];
? chains[0]
: chains[opt_block_list[0].chain_num];
if (opt_mode == MODE_COMPRESS) {
#ifdef HAVE_ENCODERS
@ -1119,11 +1067,11 @@ split_block(uint64_t *block_remaining,
++*list_pos;
// Update the filters if needed.
if (opt_block_list[*list_pos - 1].filters_index
!= opt_block_list[*list_pos].filters_index) {
const uint32_t filter_idx = opt_block_list
[*list_pos].filters_index;
const lzma_filter *next = filters[filter_idx];
if (opt_block_list[*list_pos - 1].chain_num
!= opt_block_list[*list_pos].chain_num) {
const unsigned chain_num
= opt_block_list[*list_pos].chain_num;
const lzma_filter *next = chains[chain_num];
const lzma_ret ret = lzma_filters_update(
&strm, next);
@ -1139,7 +1087,7 @@ split_block(uint64_t *block_remaining,
message_fatal(
_("Error changing to "
"filter chain %u: %s"),
(unsigned)filter_idx,
chain_num,
message_strm(ret));
}
}
@ -1278,9 +1226,10 @@ coder_normal(file_pair *pair)
ret = lzma_code(&strm, action);
// Write out if the output buffer became full.
if (strm.avail_out == 0)
if (strm.avail_out == 0) {
if (coder_write_output(pair))
break;
}
#ifdef HAVE_ENCODERS
if (ret == LZMA_STREAM_END && (action == LZMA_SYNC_FLUSH
@ -1514,9 +1463,9 @@ coder_free(void)
// in coder_set_compression_settings(). Since this is only run in
// debug mode and will be freed when the process ends anyway, we
// don't worry about freeing it.
for (uint32_t i = 1; i < ARRAY_SIZE(filters); i++) {
if (filters_used_mask & (1U << i))
lzma_filters_free(filters[i], NULL);
for (uint32_t i = 1; i < ARRAY_SIZE(chains); i++) {
if (chains_used_mask & (1U << i))
lzma_filters_free(chains[i], NULL);
}
lzma_end(&strm);

21
src/xz/coder.h
View file

@ -30,15 +30,13 @@ enum format_type {
};
/// Simple struct to track Block metadata specified through the
/// --block-list option.
/// Array of these hold the entries specified with --block-list.
typedef struct {
/// Uncompressed size of the Block
uint64_t size;
/// Index into the filters[] representing the filter chain to use
/// for this Block.
uint32_t filters_index;
/// Filter chain to use for this Block (chains[chain_num])
unsigned chain_num;
} block_list_entry;
@ -65,6 +63,19 @@ extern uint64_t opt_block_size;
/// List of block size and filter chain pointer pairs.
extern block_list_entry *opt_block_list;
/// Size of the largest Block that was specified in --block-list.
/// This is used to limit the block_size option of multithreaded encoder.
/// It's waste of memory to specify a too large block_size and reducing
/// it might even allow using more threads in some cases.
///
/// NOTE: If the last entry in --block-list is the special value of 0
/// (which gets converted to UINT64_MAX), it counts here as UINT64_MAX too.
/// This way the multithreaded encoder's Block size won't be reduced.
extern uint64_t block_list_largest;
/// Bitmask indicating which filter chains we specified in --block-list.
extern uint32_t block_list_chain_mask;
/// Set the integrity check type used when compressing
extern void coder_set_check(lzma_check check);

1
src/xz/message.h
View file

@ -115,6 +115,7 @@ extern void message_help(bool long_help);
/// Prints a help message specifically for using the --filters and
/// --filtersX command line options.
tuklib_attr_noreturn
extern void message_filters_help(void);

2
src/xz/sandbox.c
View file

@ -109,7 +109,7 @@ sandbox_enable_strict_if_allowed(int src_fd lzma_attribute((__unused__)),
}
#elif defined(HAVE_LINUX_LANDLOCK_H)
#elif defined(HAVE_LINUX_LANDLOCK)
//////////////
// Landlock //

2
src/xz/sandbox.h
View file

@ -9,7 +9,7 @@
//
///////////////////////////////////////////////////////////////////////////////
#if defined(HAVE_PLEDGE) || defined(HAVE_LINUX_LANDLOCK_H) \
#if defined(HAVE_PLEDGE) || defined(HAVE_LINUX_LANDLOCK) \
|| defined(HAVE_CAP_RIGHTS_LIMIT)
# define ENABLE_SANDBOX 1
#endif

7
src/xz/signals.c
View file

@ -49,6 +49,10 @@ signal_handler(int sig)
}
#ifdef __APPLE__
# pragma GCC diagnostic push
# pragma GCC diagnostic ignored "-Wsign-conversion"
#endif
extern void
signals_init(void)
{
@ -127,6 +131,9 @@ signals_init(void)
return;
}
#ifdef __APPLE__
# pragma GCC diagnostic pop
#endif
#ifndef __VMS

40
src/xz/xz.1
View file

@ -4,7 +4,7 @@
.\" Authors: Lasse Collin
.\" Jia Tan
.\"
.TH XZ 1 "2024-02-13" "Tukaani" "XZ Utils"
.TH XZ 1 "2024-04-08" "Tukaani" "XZ Utils"
.
.SH NAME
xz, unxz, xzcat, lzma, unlzma, lzcat \- Compress or decompress .xz and .lzma files
@ -1812,6 +1812,8 @@ and
\fB\-\-ia64\fR[\fB=\fIoptions\fR]
.TP
\fB\-\-sparc\fR[\fB=\fIoptions\fR]
.TP
\fB\-\-riscv\fR[\fB=\fIoptions\fR]
.PD
Add a branch/call/jump (BCJ) filter to the filter chain.
These filters can be used only as a non-last filter
@ -2552,7 +2554,7 @@ is run by a script or tool, for example, GNU
.RS
.PP
.nf
.ft CW
.ft CR
XZ_OPT=\-2v tar caf foo.tar.xz foo
.ft R
.fi
@ -2572,7 +2574,7 @@ scripts one may use something like this:
.RS
.PP
.nf
.ft CW
.ft CR
XZ_OPT=${XZ_OPT\-"\-7e"}
export XZ_OPT
.ft R
@ -2830,7 +2832,7 @@ if compression is successful:
.RS
.PP
.nf
.ft CW
.ft CR
xz foo
.ft R
.fi
@ -2846,7 +2848,7 @@ even if decompression is successful:
.RS
.PP
.nf
.ft CW
.ft CR
xz \-dk bar.xz
.ft R
.fi
@ -2864,7 +2866,7 @@ and 5\ MiB, respectively):
.RS
.PP
.nf
.ft CW
.ft CR
tar cf \- baz | xz \-4e > baz.tar.xz
.ft R
.fi
@ -2875,7 +2877,7 @@ to standard output with a single command:
.RS
.PP
.nf
.ft CW
.ft CR
xz \-dcf a.txt b.txt.xz c.txt d.txt.lzma > abcd.txt
.ft R
.fi
@ -2890,7 +2892,7 @@ can be used to parallelize compression of many files:
.RS
.PP
.nf
.ft CW
.ft CR
find . \-type f \e! \-name '*.xz' \-print0 \e
| xargs \-0r \-P4 \-n16 xz \-T1
.ft R
@ -2930,7 +2932,7 @@ after compressing multiple files:
.RS
.PP
.nf
.ft CW
.ft CR
xz \-\-robot \-\-list *.xz | awk '/^totals/{print $5\-$4}'
.ft R
.fi
@ -2950,7 +2952,7 @@ option:
.RS
.PP
.nf
.ft CW
.ft CR
if ! eval "$(xz \-\-robot \-\-version 2> /dev/null)" ||
[ "$XZ_VERSION" \-lt 50000002 ]; then
echo "Your xz is too old."
@ -2966,7 +2968,7 @@ but if a limit has already been set, don't increase it:
.RS
.PP
.nf
.ft CW
.ft CR
NEWLIM=$((123 << 20))\ \ # 123 MiB
OLDLIM=$(xz \-\-robot \-\-info\-memory | cut \-f3)
if [ $OLDLIM \-eq 0 \-o $OLDLIM \-gt $NEWLIM ]; then
@ -3019,7 +3021,7 @@ can be modified to use a bigger dictionary:
.RS
.PP
.nf
.ft CW
.ft CR
xz \-\-lzma2=preset=1,dict=32MiB foo.tar
.ft R
.fi
@ -3047,7 +3049,7 @@ would use:
.RS
.PP
.nf
.ft CW
.ft CR
xz \-vv \-\-lzma2=dict=192MiB big_foo.tar
.ft R
.fi
@ -3077,7 +3079,7 @@ using about 100\ KiB of memory.
.RS
.PP
.nf
.ft CW
.ft CR
xz \-\-check=crc32 \-\-lzma2=preset=6e,dict=64KiB foo
.ft R
.fi
@ -3105,7 +3107,7 @@ slightly (like 0.1\ %) smaller file than
.RS
.PP
.nf
.ft CW
.ft CR
xz \-\-lzma2=preset=6e,pb=0,lc=4 source_code.tar
.ft R
.fi
@ -3118,7 +3120,7 @@ using the x86 BCJ filter:
.RS
.PP
.nf
.ft CW
.ft CR
xz \-\-x86 \-\-lzma2 libfoo.so
.ft R
.fi
@ -3153,7 +3155,7 @@ to LZMA2 to accommodate the three-byte alignment:
.RS
.PP
.nf
.ft CW
.ft CR
xz \-\-delta=dist=3 \-\-lzma2=pb=0 foo.tiff
.ft R
.fi
@ -3174,8 +3176,8 @@ have the same number of bytes per pixel.
.BR bzip2 (1),
.BR 7z (1)
.PP
XZ Utils: <https://xz.tukaani.org/xz-utils/>
XZ Utils: <https://tukaani.org/xz/>
.br
XZ Embedded: <https://xz.tukaani.org/xz-embedded/>
XZ Embedded: <https://tukaani.org/xz/embedded.html>
.br
LZMA SDK: <https://7-zip.org/sdk.html>

4
src/xzdec/xzdec.1
View file

@ -2,7 +2,7 @@
.\"
.\" Author: Lasse Collin
.\"
.TH XZDEC 1 "2024-01-19" "Tukaani" "XZ Utils"
.TH XZDEC 1 "2024-04-08" "Tukaani" "XZ Utils"
.SH NAME
xzdec, lzmadec \- Small .xz and .lzma decompressors
.SH SYNOPSIS
@ -141,4 +141,4 @@ decompressor, consider using XZ Embedded.
.SH "SEE ALSO"
.BR xz (1)
.PP
XZ Embedded: <https://xz.tukaani.org/xz-embedded/>
XZ Embedded: <https://tukaani.org/xz/embedded.html>

58
src/xzdec/xzdec.c
View file

@ -24,14 +24,19 @@
# include <sys/capsicum.h>
#endif
#ifdef HAVE_LINUX_LANDLOCK_H
#ifdef HAVE_LINUX_LANDLOCK
# include <linux/landlock.h>
# include <sys/prctl.h>
# include <sys/syscall.h>
# ifdef LANDLOCK_ACCESS_NET_BIND_TCP
# define LANDLOCK_ABI_MAX 4
# else
# define LANDLOCK_ABI_MAX 3
# endif
#endif
#if defined(HAVE_CAP_RIGHTS_LIMIT) || defined(HAVE_PLEDGE) \
|| defined(HAVE_LINUX_LANDLOCK_H)
|| defined(HAVE_LINUX_LANDLOCK)
# define ENABLE_SANDBOX 1
#endif
@ -313,10 +318,12 @@ sandbox_enter(int src_fd)
STDIN_FILENO, cap_rights_clear(&rights)))
goto error;
if (cap_rights_limit(STDOUT_FILENO, cap_rights_init(&rights, CAP_WRITE)))
if (cap_rights_limit(STDOUT_FILENO, cap_rights_init(&rights,
CAP_WRITE)))
goto error;
if (cap_rights_limit(STDERR_FILENO, cap_rights_init(&rights, CAP_WRITE)))
if (cap_rights_limit(STDERR_FILENO, cap_rights_init(&rights,
CAP_WRITE)))
goto error;
#elif defined(HAVE_PLEDGE)
@ -325,17 +332,23 @@ sandbox_enter(int src_fd)
goto error;
(void)src_fd;
#elif defined(HAVE_LINUX_LANDLOCK_H)
#elif defined(HAVE_LINUX_LANDLOCK)
int landlock_abi = syscall(SYS_landlock_create_ruleset,
(void *)NULL, 0, LANDLOCK_CREATE_RULESET_VERSION);
if (landlock_abi > 0) {
// We support ABI versions 1-3.
if (landlock_abi > 3)
landlock_abi = 3;
if (landlock_abi > LANDLOCK_ABI_MAX)
landlock_abi = LANDLOCK_ABI_MAX;
const struct landlock_ruleset_attr attr = {
.handled_access_fs = (1ULL << (12 + landlock_abi)) - 1
.handled_access_fs = (1ULL
<< (12 + my_min(3, landlock_abi))) - 1,
# if LANDLOCK_ABI_MAX >= 4
.handled_access_net = landlock_abi < 4 ? 0 :
(LANDLOCK_ACCESS_NET_BIND_TCP
| LANDLOCK_ACCESS_NET_CONNECT_TCP),
# endif
};
const int ruleset_fd = syscall(SYS_landlock_create_ruleset,
@ -351,6 +364,7 @@ sandbox_enter(int src_fd)
}
(void)src_fd;
#else
# error ENABLE_SANDBOX is defined but no sandboxing method was found.
#endif
@ -367,6 +381,7 @@ sandbox_enter(int src_fd)
if (errno == ENOSYS)
return;
#endif
my_errorf("Failed to enable the sandbox");
exit(EXIT_FAILURE);
}
@ -389,9 +404,15 @@ main(int argc, char **argv)
}
#endif
#ifdef HAVE_LINUX_LANDLOCK_H
// Prevent the process from gaining new privileges. The return
// is ignored to keep compatibility with old kernels.
#ifdef HAVE_LINUX_LANDLOCK
// Prevent the process from gaining new privileges. This must be done
// before landlock_restrict_self(2) but since we will never need new
// privileges, this call can be done here already.
//
// This is supported since Linux 3.5. Ignore the return value to
// keep compatibility with old kernels. landlock_restrict_self(2)
// will fail if the no_new_privs attribute isn't set, thus if prctl()
// fails here the error will still be detected when it matters.
(void)prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
#endif
@ -438,19 +459,18 @@ main(int argc, char **argv)
}
}
#ifdef ENABLE_SANDBOX
// Enable the sandbox for the last file. When the
// strict sandbox is enabled the process can no
// longer open additional files. It is likely that
// the most common way to use xzdec is to
// decompress a single file, so this fully protects
// most use cases.
// Enable the strict sandbox for the last file.
// Then the process can no longer open additional
// files. The typical xzdec use case is to decompress
// a single file so this way the strictest sandboxing
// is used in most cases.
if (optind == argc - 1)
sandbox_enter(fileno(src_file));
#endif
uncompress(&strm, src_file, src_name);
if (src_file != stdin)
fclose(src_file);
(void)fclose(src_file);
} while (++optind < argc);
}