Commit graph

58 commits

Author SHA1 Message Date
Ka Ho Ng fb860ed0c5 libfetch: parse scheme://domain:/ correctly
This improves URL-parsing compability with cURL, and unbreaks parsing of
similar kinds of URLs after commit 8d9de5b10a.

Sponsored by:	Juniper Networks, Inc.
Reviewed by:	des
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D44493
2024-03-25 16:11:42 -04:00
Warner Losh a2f733abcf lib: Automated cleanup of cdefs and other formatting
Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by:		Netflix
2023-11-26 22:23:59 -07:00
Warner Losh 1d386b48a5 Remove $FreeBSD$: one-line .c pattern
Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
2023-08-16 11:54:42 -06:00
Ed Maste 83372bda16 libfetch: disallow invalid escape sequences
Per RFC1738 escape is "% hex hex"; other sequences do not form a valid URL.

Suggested by:	Matthew Dillon
Reviewed by:	Matthew Dillon
MFC after:	1 week
2020-02-05 16:55:00 +00:00
Gordon Tetlow 6fb3f9944f Fix urldecode buffer overrun.
Reported by:	Duncan Overbruck
Security:	CVE-2020-7450
2020-01-28 18:37:18 +00:00
Dag-Erling Smørgrav 8d9de5b10a Improve URL parsing. In particular, convert scheme and host to lowercase.
MFC after:	1 week
2018-11-27 10:45:14 +00:00
Dag-Erling Smørgrav 5f04ebd4d3 Fix an inverted conditional in the netrc code, which would ignore the
value of $HOME and always use the home directory from the passwd
database, unless $HOME was unset, in which case it would use (null).

While there, clean up handling of netrcfd and add debugging aids.

MFC after:	3 weeks
2018-05-29 13:07:36 +00:00
Dag-Erling Smørgrav c5712d6da1 Use __VA_ARGS__ to simplify the DEBUG macro.
MFC after:	3 weeks
2018-05-29 10:28:20 +00:00
Pedro F. Giffuni 5e53a4f90f lib: further adoption of SPDX licensing ID tags.
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
2017-11-26 02:00:33 +00:00
Dag-Erling Smørgrav 08a49957b3 r308996 broke IP literals by assuming that a colon could only occur as
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals correctly, and
remove similar code elsewhere to avoid passing unbracketed literals to
fetch_resolve().  Remove #ifdef INET6 so we still parse IP literals
correctly even if we do not have the ability to connect to them.

While there, fix an off-by-one error which caused HTTP 400 errors to be
misinterpreted as redirects.

PR:		217723
MFC after:	1 week
Reported by:	bapt, bz, cem, ngie
2017-03-17 14:18:52 +00:00
Baptiste Daroussin 5d1ce10bee Properly initialize netrcfd in fetchParseURL
This fixes ftp with fetch(1) which was broken after r313974

Submitted by:	dim
Reported by:	olivier
Pointyhat to:	bapt
2017-03-03 12:51:16 +00:00
Baptiste Daroussin d8713bf361 Add a file descriptor in struct url for netrc
When using libfetch in an application that drops privileges when fetching
like pkg(8) then user complain because the application does not read anymore
${HOME}/.netrc. Now a caller can prepare a fd to the said file and manually
assign it to the structure.

It is also a first step to allow to capsicumize libfetch applications

Reviewed by:	allanjude, des
Approved by:	des
Differential Revision:	https://reviews.freebsd.org/D9678
2017-02-20 00:14:31 +00:00
Tim Kientzle 9bc22394d8 Fix -Wunsequenced warning.
Submitted by:	dt71@gmx.com
2013-06-29 15:51:27 +00:00
Ed Maste 0fa391997a Support percent-encoded user and password
RFC 1738 specifies that any ":", "@", or "/" within a user name or
password in a URL is percent-encoded, to avoid ambiguity with the use
of those characters as URL component separators.

Reviewed by:    rstone@
MFC after:	1 month
2012-04-11 15:42:02 +00:00
Dag-Erling Smørgrav 578153f1ba latin1 -> utf8 2011-10-19 11:43:51 +00:00
Dag-Erling Smørgrav 6337341d81 Update copyright dates and strip my middle name. 2011-09-27 18:57:26 +00:00
Murray Stokely e8f612b687 1. Update fetch to consistently return 1 on error, as the man page states,
rather than usually returning 1 but in a few instances using a sysexits(3)
   return value.

2. Remove a few unused variables from libfetch.

PR:		docs/122470  (1, only)
Reviewed by:	des
2008-12-17 18:00:18 +00:00
Dag-Erling Smørgrav facd982794 As several people pointed out, I did all the ctype casts the wrong
way (not for the first time...)

Noticed by:	bde, ru ++
MFC after:	1 week
2007-12-19 00:26:36 +00:00
Dag-Erling Smørgrav 836e34eeae Old patch I had lying around: correctly cast the argument to is*().
IWBNI gcc could warn about this the way it warns about printf() abuse.

MFC after:	1 week
2007-12-18 10:41:12 +00:00
Dag-Erling Smørgrav a1b37df2d7 Clean up namespace violations.
MFC after:	1 week
2007-12-14 10:26:58 +00:00
Dag-Erling Smørgrav 2cbbf9dac9 Update copyright years. 2004-09-21 18:35:21 +00:00
Dag-Erling Smørgrav 930105c1e9 style(9): add parentheses to sizeof even when not strictly required.
MFC after:	3 days
2003-01-28 08:04:40 +00:00
Dag-Erling Smørgrav 02e6bec10b Initialization paranoia. 2003-01-22 17:53:15 +00:00
Dag-Erling Smørgrav 111e251009 Add SSL support + slight cleanup.
Submitted by:	Henry Whincup <henry@techiebod.com> (in principle)
2002-06-05 12:46:36 +00:00
Dag-Erling Smørgrav e19e6098b3 Reindent, and add parentheses to return statements. Some functions in
ftp.c and http.c now have exceedingly long lines due to deep nesting;
this will be corrected by reorganizing the code in a later revision.
2002-02-05 22:13:51 +00:00
Dag-Erling Smørgrav 551858f0b0 Eliminate a redundant loop. 2002-02-05 21:32:16 +00:00
Dag-Erling Smørgrav f67efa37d6 Remove VT100 escapes from debugging messages now that they're enabled by
default.

PR:		32988
MFC after:	3 days
2002-01-01 14:48:09 +00:00
Dag-Erling Smørgrav 7eb2f34d73 Introduce a fetchDebug global. Change the DEBUG macro so it only runs the
debugging code if fetchDebug is set.

PR:		bin/32615
MFC after:	1 week
2001-12-09 15:05:19 +00:00
Matthew Dillon cecb889f1d Add __FBSDID()s to libfetch 2001-09-30 21:36:09 +00:00
Stefan Eßer 73b3e4df87 The fix for schemeless and hostless URLs (rev. 1.27) broke the schemeless
proxy specification, which seems to be valid according to the man page.

Change the logic to consider "hostname:port" a hostname and port instead
of a file URL.

Approved by:	des
2001-07-28 21:28:14 +00:00
Dag-Erling Smørgrav 5b2ad516e9 Unbreak parsing URLs that have a host part but no document part.
While we're here, fix a snprintf() usage warning.
2001-07-17 20:22:33 +00:00
Dag-Erling Smørgrav ab39353eef Handle shemeless, hostless URLs correctly. 2001-07-08 15:59:15 +00:00
Dag-Erling Smørgrav 6490b215b3 Add rudimentary support for an authentication callback function. 2001-05-26 19:37:15 +00:00
Archie Cobbs 38c7e4a631 Apply 'const' liberally.
Fix some other minor glitches.
2001-04-24 00:06:21 +00:00
Dag-Erling Smørgrav 1ba8497660 Avoid a segfault (due to an unitialized pointer) when parsing URLs that have
no scheme or host part.
2000-12-13 11:21:09 +00:00
Dag-Erling Smørgrav f9c2053bca Clean up the whitespace encoding code. 2000-12-01 11:04:57 +00:00
Dag-Erling Smørgrav d74a913b68 Use CHECK_FLAG 2000-10-29 15:56:10 +00:00
Dag-Erling Smørgrav 23fe6d7a4c If the scheme is HTTP or HTTPS, percent-escape whitespace in the document
part.

Submitted by:	green
2000-10-21 14:58:18 +00:00
Dag-Erling Smørgrav 59769ab192 Relax URL syntax so that schemeless URLs are supported. 2000-10-12 22:07:49 +00:00
Dag-Erling Smørgrav dfe7c55fcc The decision to use an HTTP proxy for FTP documents belongs in the FTP code 2000-07-21 10:46:22 +00:00
Dag-Erling Smørgrav 1a5faa1061 Introduce fetchXGet*(), which combine the functionalities of fetchGet*() and
fetchStat*().  In most cases, either fetchGet*() or fetchXGet*() is a wrapper
around the other; in all cases, calling fetchGet*() is identical to calling
fetchXGet*() with the second argument set to NULL.
2000-07-17 21:25:00 +00:00
Dag-Erling Smørgrav a1bb3f482a Introduce the (undocumented) variable fetchRestartCalls, which controls whether
or not interrupted system calls will be restarted. This fixes a bug where
fetch(1) would hang (potentially forever) if a server stopped responding,
because the signal handler would absorb the user's efforts to interrupt the
transfer.
2000-07-17 20:49:39 +00:00
Dag-Erling Smørgrav 9a964d6a82 Add (and document) fetchMakeURL() 2000-05-25 16:46:09 +00:00
Dag-Erling Smørgrav ba101983d5 Dump com_err, it's a stinking crock of shit. 2000-05-25 16:24:31 +00:00
Hajimu UMEMOTO 28c645cf6a IPv6 support. 2000-05-20 18:23:51 +00:00
Hajimu UMEMOTO c97925ad4e Make HTTP_PROXY work for FTP.
Reported by:	Ben Smithurst <ben@scientia.demon.co.uk>
Reviewed by:	des
2000-05-15 09:05:36 +00:00
Dag-Erling Smørgrav 60245e42b0 Reorganize some of the http code and split it into more functions.
Implement fetchStatHTTP().
Unbungle struct url, and add fetchFreeURL().
Document it.
2000-05-11 13:31:02 +00:00
Dag-Erling Smørgrav 807c941c90 Use sizeof more consistently. 2000-05-07 20:02:52 +00:00
Dag-Erling Smørgrav fc6e9e6539 More old uncommitted patches: implement timeouts at the protocol level.
Currently only supported for ftp connections.
2000-01-07 12:58:40 +00:00
Peter Wemm 7f3dea244c $Id$ -> $FreeBSD$ 1999-08-28 00:22:10 +00:00