Commit graph

297 commits

Author SHA1 Message Date
Eugene Grosbein 7a6309fdc7 fetch(1): suppress "Not Modified" in quiet mode
"fetch -qi" should skip printing "Not Modified" for successful
http(s) request. Still print it by default (v_level == 1).

MFC after:	1 week
2024-09-07 09:14:23 +07:00
Warner Losh e9ac41698b Remove residual blank line at start of Makefile
Some checks are pending
Cross-build Kernel / ${{ matrix.target_arch }} ${{ matrix.os }} (${{ matrix.compiler }}) (clang-14, /usr/lib/llvm-14/bin, ubuntu-22.04, bmake libarchive-dev clang-14 lld-14, arm64, aarch64) (push) Waiting to run
Cross-build Kernel / ${{ matrix.target_arch }} ${{ matrix.os }} (${{ matrix.compiler }}) (clang-14, /usr/lib/llvm-14/bin, ubuntu-22.04, bmake libarchive-dev clang-14 lld-14, amd64, amd64) (push) Waiting to run
Cross-build Kernel / ${{ matrix.target_arch }} ${{ matrix.os }} (${{ matrix.compiler }}) (clang-13, /opt/homebrew/opt/llvm@13/bin, macos-latest, bmake libarchive llvm@13, arm64, aarch64) (push) Waiting to run
Cross-build Kernel / ${{ matrix.target_arch }} ${{ matrix.os }} (${{ matrix.compiler }}) (clang-13, /opt/homebrew/opt/llvm@13/bin, macos-latest, bmake libarchive llvm@13, amd64, amd64) (push) Waiting to run
Cross-build Kernel / ${{ matrix.target_arch }} ${{ matrix.os }} (${{ matrix.compiler }}) (clang-12, /usr/lib/llvm-12/bin, ubuntu-20.04, bmake libarchive-dev clang-12 lld-12, arm64, aarch64) (push) Waiting to run
Cross-build Kernel / ${{ matrix.target_arch }} ${{ matrix.os }} (${{ matrix.compiler }}) (clang-12, /usr/lib/llvm-12/bin, ubuntu-20.04, bmake libarchive-dev clang-12 lld-12, amd64, amd64) (push) Waiting to run
This is a residual of the $FreeBSD$ removal.

MFC After: 3 days (though I'll just run the command on the branches)
Sponsored by: Netflix
2024-07-15 16:43:39 -06:00
Warner Losh 5e3934b15a usr.bin: Automated cleanup of cdefs and other formatting
Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by:		Netflix
2023-11-26 22:24:01 -07:00
Dag-Erling Smørgrav 2821a7498f libfetch, fetch: Stop recommending the use of ca_root_nss.
MFC after:	3 days
Reviewed by:	kevans, emaste
Differential Revision:	https://reviews.freebsd.org/D42119
2023-10-08 06:35:15 +02:00
Warner Losh fa9896e082 Remove $FreeBSD$: two-line nroff pattern
Remove /^\.\\"\n\.\\"\s*\$FreeBSD\$$\n/
2023-08-16 11:55:10 -06:00
Warner Losh d0b2dbfa0e Remove $FreeBSD$: one-line sh pattern
Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/
2023-08-16 11:55:03 -06:00
Warner Losh 1d386b48a5 Remove $FreeBSD$: one-line .c pattern
Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
2023-08-16 11:54:42 -06:00
Simon J. Gerraty d9a4274795 Update/fix Makefile.depend for userland 2023-04-18 17:14:23 -07:00
Stefan Eßer 32066c96fa fetch: make -S argument accept values > 2GB
Use strtoll() to parse the argument of the -S option.

FreeBSD has supported 64 bit file offsets for more than 25 years on
all architectures and off_t is a 64 bit integer type for that reason.

While strtol() returns a 64 bit value on 64 LP64 architectures, it
is limit to 32 bit on e.g. i386. The strtoll() function returns a 64
but result on all supported architectures and therefore supports the
possible file lengths and file offsets on 32 bit archtectures.

Reported by:	antoine
MFC after:	3 days
2022-02-20 15:24:43 +01:00
Eugene Grosbein 85f15576b4 fetch(1): more fixes for soft failure handling
Fix logic error introduced in my commit
bf599c03f0

Also, authorization errors should not be considered as soft failures.
2022-01-25 12:48:28 +07:00
Eugene Grosbein 08a2504a20 fetch(1): fix error in previous commit
strncmp() compares terminating zero and sizeof() includes it.
Un-obsfuscate the code and show what it is indended to do.
2022-01-24 15:07:18 +07:00
Eugene Grosbein bf599c03f0 fetch(1): do not consider HTTP 5XX errors as soft failures
This change fixes "fetch -a" looping forever on "502 Bad gateway"
error and similar.

MFC after:	1 month
2022-01-24 14:38:26 +07:00
Eugene Grosbein a4efbe0d6d fetch(1): correct progress accounting after previous commit
MFC after:	1 month
2022-01-24 11:17:24 +07:00
Eugene Grosbein e3bad5f7aa fetch(1): process truncated transfer as soft failure
Let "fetch -a" resume truncated transfer automatically
perform another attempt if it obtained some new data in previous one
making progress.

This makes it more robust against frequent but transient network failures.
For example:

=> sqlite-src-3370200.zip doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://www.sqlite.org/2022/sqlite-src-3370200.zip
sqlite-src-3370200.zip                          3% of   12 MB   45 kBps 04m24s
fetch: sqlite-src-3370200.zip appears to be truncated: 524288/13145234 bytes
sqlite-src-3370200.zip                         10% of   12 MB   67 kBps 02m56s
fetch: sqlite-src-3370200.zip appears to be truncated: 1327104/13145234 bytes
sqlite-src-3370200.zip                         28% of   12 MB  123 kBps 01m14s
fetch: sqlite-src-3370200.zip appears to be truncated: 3735552/13145234 bytes
sqlite-src-3370200.zip                         54% of   12 MB  253 kBps    24s
fetch: sqlite-src-3370200.zip appears to be truncated: 7176192/13145234 bytes
sqlite-src-3370200.zip                         62% of   12 MB   90 kBps    55s
fetch: sqlite-src-3370200.zip appears to be truncated: 8241152/13145234 bytes
sqlite-src-3370200.zip                         82% of   12 MB  113 kBps    20s
fetch: sqlite-src-3370200.zip appears to be truncated: 10862592/13145234 bytes
sqlite-src-3370200.zip                                  12 MB  185 kBps    12s
===> Fetching all distfiles required by sqlite3-3.37.2,1 for building

MFC after:	1 month
2022-01-24 11:09:37 +07:00
Emmanuel Vadot 13ef8134ef pkgbase: Create a FreeBSD-fetch package
It's useful for small image to fetch some data but we don't want to
install utilities nor bloat runtime.

MFC after:	2 weeks
Sponsored by:	Beckhoff Automation GmbH & Co. KG
Differential Revision:	https://reviews.freebsd.org/D33463
2021-12-21 10:17:46 +01:00
Fernando Apesteguía 86e149e16e fetch(1): Fix style
Fix style problems introduced in r367141:

* s/Ql/Pa/ for file names
* Break line properly in macro
* Properly generate back slash character

Not bumping .Dd

Reported by:	xtouqh@icloud.com, steffen@sdaoden.eu
Approved by:	0mp@
Differential Revision:		https://reviews.freebsd.org/D26946
2020-10-30 09:48:41 +00:00
Fernando Apesteguía 538d212d13 fetch(1): Add EXAMPLES section
Add a few examples covering flags: 1, R, a, o, q, r, s, v

Approved by:	manpages (bcr@)
Differential Revision:		https://reviews.freebsd.org/D26946
2020-10-29 18:34:47 +00:00
Xin LI 941791759c Don't explicitly specify c99 or gnu99 as the default is now gnu99.
MFC after:	2 weeks
2020-08-17 05:57:02 +00:00
Dag-Erling Smørgrav 6434dda6e3 Through a combination of insufficient variable initialization and
imprudent reuse of static buffers, the end-of-transfer statistics
displayed when stdout is not a tty always ended up as 0 B / 0 Bps.
Reorganize the code to use caller-provided buffers, tweak the ETA
display a bit, and reduce the visual differences between the tty and
non-tty end-of-transfer displays.

PR:		202424
Approved by:	re (gjb@)
2018-09-10 19:39:20 +00:00
Dag-Erling Smørgrav deb1ff232d Support If-Modified-Since for https as well as http.
PR:		224426
Submitted by:	zsnafzig@edu.uwaterloo.ca
MFC after:	1 week
2018-05-12 17:04:40 +00:00
Benjamin Kaduk b24eca8b54 Fix spelling of --output long option in fetch.1
PR:		228017
Reported by:	rff1917@yahoo.com
2018-05-06 15:59:03 +00:00
Pedro F. Giffuni 1de7b4b805 various: general adoption of SPDX licensing ID tags.
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.
2017-11-27 15:37:16 +00:00
Bryan Drewery ea825d0274 DIRDEPS_BUILD: Update dependencies.
Sponsored by:	Dell EMC Isilon
2017-10-31 00:07:04 +00:00
Edward Tomasz Napierala 43a623c82f Add phttpget(8) .Xr to fetch(1).
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2016-03-29 11:56:00 +00:00
Michael Gmelin 1ec923fd6d Update fetch.1 and fetch.3 to reflect libfetch's actual use of CA bundles
Reviewed by:	wblock
Approved by:	wblock
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D5558
2016-03-19 11:55:21 +00:00
Simon J. Gerraty ccfb965433 Add META_MODE support.
Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision:       D2796
Reviewed by: brooks imp
2015-06-13 19:20:56 +00:00
Simon J. Gerraty 44d314f704 dirdeps.mk now sets DEP_RELDIR 2015-06-08 23:35:17 +00:00
Simon J. Gerraty 98e0ffaefb Merge sync of head 2015-05-27 01:19:58 +00:00
Jung-uk Kim 01ed3ca3de Remove defunct SSLv2 support from fetch(1) and fetch(3). 2015-03-25 18:56:36 +00:00
Baptiste Daroussin 3e11bd9e2a Convert to usr.bin/ to LIBADD
Reduce overlinking
2014-11-25 14:29:10 +00:00
Simon J. Gerraty ee7b0571c2 Merge head from 7/28 2014-08-19 06:50:54 +00:00
Baptiste Daroussin 2b7af31cf5 use .Mt to mark up email addresses consistently (part3)
PR:		191174
Submitted by:	Franco Fichtner  <franco at lastsummer.de>
2014-06-23 08:23:05 +00:00
Simon J. Gerraty fae50821ae Updated dependencies 2014-05-16 14:09:51 +00:00
Simon J. Gerraty cc3f4b9965 Merge from head 2014-05-08 23:54:15 +00:00
Warner Losh c6063d0da8 Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.
2014-05-06 04:22:01 +00:00
Simon J. Gerraty 3b8f084595 Merge head 2014-04-28 07:50:45 +00:00
Dag-Erling Smørgrav c2ea176dac Increase the default (and minimum) buffer size from 4 kB to 16 kB. Also,
propagate the buffer size to libc, which uses a 1 kB buffer by default,
negating any hypothetical benefit of increasing fetch(1)'s buffer size.

MFC after:	3 days
2014-01-28 14:32:04 +00:00
Dag-Erling Smørgrav 2043423cd1 whitespace and bump copyright 2014-01-28 14:29:24 +00:00
Simon J. Gerraty 34b33809b7 Updated dependencies 2013-10-13 00:24:00 +00:00
Simon J. Gerraty d1d0158641 Merge from head 2013-09-05 20:18:59 +00:00
Dag-Erling Smørgrav 1453595f49 Include an Accept header in requests.
PR:		kern/180917
MFC after:	1 week
2013-07-30 13:07:55 +00:00
Dag-Erling Smørgrav cb67f01df2 Add Michael Gmelin's name to the copyright statement. 2013-07-30 13:05:51 +00:00
Joel Dahl b5a4266b48 Remove EOL whitespace. 2013-07-28 06:16:12 +00:00
Dag-Erling Smørgrav dcd47379ff Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR:		kern/175514
Submitted by:	Michael Gmelin <freebsd@grem.de>
MFC after:	1 week
2013-07-26 15:53:43 +00:00
Dag-Erling Smørgrav 011a8335f8 mdoc nit 2013-07-21 07:24:25 +00:00
Eitan Adler a96469b791 Correct check to see if file changes.
Reported by:	swildner@DragonFlyBSD.org
Reviewed by:	des
2013-06-02 18:10:07 +00:00
Simon J. Gerraty f5f7c05209 Updated dependencies 2013-02-16 01:23:54 +00:00
David E. O'Brien d9a447559b Sync with HEAD. 2013-02-08 16:10:16 +00:00
Andre Oppermann 162361c198 Fix bandwidth reporting when doing a restarted download with "-r".
The offset is already accounted for in xs->lastrcvd and doesn't
have to be subtracted again.

Reported by:	Florian Smeets <flo@smeets.im>
Submitted by:	Mateusz Guzik <mjguzik@gmail.com>
Tested by:	Florian Smeets <flo@smeets.im>
MFC after:	1 week
2012-12-09 22:54:03 +00:00
Eitan Adler 2214137ab1 Add check for failure of mkstemp and setenv.
Reviewed by:	des
Approved by:	cperciva (implicit)
Obtained from:	DragonFlyBSD
MFC after:	1 week
2012-12-08 18:41:16 +00:00