a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals correctly, and
remove similar code elsewhere to avoid passing unbracketed literals to
fetch_resolve(). Remove #ifdef INET6 so we still parse IP literals
correctly even if we do not have the ability to connect to them.
While there, fix an off-by-one error which caused HTTP 400 errors to be
misinterpreted as redirects.
PR: 217723
MFC after: 1 week
Reported by: bapt, bz, cem, ngie
already complete.
Since 416 is an error code, any Content-Range header in the response
would refer to the error message, not the requested document, so
relying on the value of size when we know we got a 416 is wrong.
Instead, just verify that offset == 0 and assume that we've reached
the end of the document (if offset > 0, we did not request a range,
and the server is screwing with us). Note that we cannot distinguish
between reaching the end and going past it, but that is a flaw in the
protocol, not in the code, so we just have to assume that the caller
knows what it's doing. A smart caller would request an offset
slightly before what it believes is the end and compare the result to
what is already in the file.
PR: 212065
Reported by: mandree
MFC after: 3 weeks
When using libfetch in an application that drops privileges when fetching
like pkg(8) then user complain because the application does not read anymore
${HOME}/.netrc. Now a caller can prepare a fd to the said file and manually
assign it to the structure.
It is also a first step to allow to capsicumize libfetch applications
Reviewed by: allanjude, des
Approved by: des
Differential Revision: https://reviews.freebsd.org/D9678
in verbose mode, and did not handle 308 at all.
r241840 (PR 172451) added support for 308, but with the same bug.
Correctly handle both by recognizing them as redirects in all places
where we check the HTTP result code.
PR: 112515 173451 209546
Submitted by: novel@
MFC after: 1 week
setlocale() so that static analyzers know that the string is NUL
terminated. This was causing a false positive in Coverity even
though the longest string returned by setlocale() is ENCODING_LEN
(31) and we are copying into a 64 byte buffer. This change is also
a bit of an optimization since we don't need the strncpy() feature
of padding the rest of the destination buffer with NUL characters.
Reported by: Coverity
CID: 974654
These are no longer needed after the recent 'beforebuild: depend' changes
and hooking DIRDEPS_BUILD into a subset of FAST_DEPEND which supports
skipping 'make depend'.
Sponsored by: EMC / Isilon Storage Division
to the end of the function, but did not remove a fetch_close() call which
was made redundant by the one in the shared error-handling code.
PR: 206774
Submitted by: Christian Heckendorf <heckendorfc@gmail.com>
MFC after: 3 days
Prior to this patch, unless SSL_CA_CERT_FILE is set in the environment,
libfetch will set the CA file to "/usr/local/etc/cert.pem" if it exists,
and to "/etc/ssl/cert.pem" otherwise. This has the consequence of
masking SSL_CA_CERT_PATH, because OpenSSL will ignore the CA path if a CA
file is set but fails to load (see X509_STORE_load_locations()).
While here, fall back to OpenSSL defaults if neither SSL_CA_CERT_FILE nor
SSL_CA_CERT_PATH are set in the environment, and if neither of the
libfetch default CA files exists.
PR: 193871
Submitted by: John W. O'Brien <john@saltant.com>
Approved by: des
MFC after: 1 week
risk leaving the connection in an indeterminate state if the server fails
to send a chunk delimiter. Depending on the application and on the sizes
of the preceding chunks, the result can be anything from missing data to a
segfault. With this patch, it will be reported as a protocol error.
PR: 204771
MFC after: 1 week
This both avoids some dependencies on xinstall.host and allows
bootstrapping on older releases to work due to lack of at least 'install -l'
support.
Sponsored by: EMC / Isilon Storage Division
- If the proxy returns a non-200 result, set the error code accordingly
so the caller / user gets a somewhat meaningful error message.
- Consume and discard any HTTP response header following the result line.
PR: 194483
Tested by: Fabian Keil <fk@fabiankeil.de>
MFC after: 1 week
lib/libfetch/http.c:1628:26: error: address of array 'purl->user'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.user = purl->user ?
~~~~~~^~~~ ~
lib/libfetch/http.c:1630:30: error: address of array 'purl->pwd'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.password = purl->pwd?
~~~~~~^~~~
lib/libfetch/http.c:1657:25: error: address of array 'url->user'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.user = url->user ?
~~~~~^~~~ ~
lib/libfetch/http.c:1659:29: error: address of array 'url->pwd'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.password = url->pwd ?
~~~~~^~~ ~
lib/libfetch/http.c:1669:25: error: address of array 'url->user'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.user = url->user ?
~~~~~^~~~ ~
lib/libfetch/http.c:1671:29: error: address of array 'url->pwd'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.password = url->pwd ?
~~~~~^~~ ~
Since url->user and url->pwd are arrays, they can never be NULL, so the
checks can be removed.
Reviewed by: bapt
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D2673
Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.
Still need to add real targets under targets/ to build packages.
Differential Revision: D2796
Reviewed by: brooks imp
disabled everything except TLS 1.0. Replace it with a more carefully
wrought patch:
- Switch the default for SSLv3 from on to off
- Add environment variables to control TLS 1.1 and 1.2
- In verbose mode, report which version is used
- Update the man page to reflect these changes.
MFC after: 1 week
The standard states that GMT must be used, but that UTC is equivalent. Still
parse UTC as otherwise this causes problems for pkg(8). It will refetch
the repository every time 'pkg update' or other remote operations
are used behind these proxies.
RFC2616: "All HTTP date/time stamps MUST be represented in Greenwich Mean
Time (GMT), without exception. For the purposes of HTTP, GMT is exactly equal
to UTC (Coordinated Universal Time).""
Approved by: bapt (mentor)
Reviewed by: des, peter
Sponsored by: EMC / Isilon Storage Division
MFC after: 1 week
known in advance, or where the caller doesn't care and just keeps
reading until it hits EOF.
In fetch_read(): the socket is non-blocking, so read() will return 0
on EOF, and -1 (errno == EAGAIN) when the connection is still open but
there is no data waiting. In the first case, we should immediately
return 0. The EINTR case was also broken, although not in a way that
matters.
In fetch_writev(): use timersub() and timercmp() as in fetch_read().
In http_fillbuf(): set errno to a sensible value when an invalid chunk
header is encountered.
In http_readfn(): as in fetch_read(), a zero return from down the
stack indicates EOF, not an error. Furthermore, when io->error is
EINTR, clear it (but no errno) before returning so the caller can
retry after dealing with the interrupt.
MFC after: 3 days
simply not trying to return exactly what the caller asked for - just
return whatever we got and let the caller be the judge of whether it
was enough. If an error occurs or the connection times out after we
already received some data, return a short read, under the assumption
that the next call will fail or time out before we read anything.
As it turns out, none of the code that calls fetch_read() assumes an
all-or-nothing result anyway, except for a couple of lines where we
read the CR LF at the end of a hunk in HTTP hunked encoding, so the
changes outside of fetch_read() and http_readfn() are minimal.
While there, replace select(2) with poll(2).
MFC after: 3 days
SSL_set_tlsext_host_name(3) internally does not modify the host buffer
pased to it. So it is safe to DECONST the struct url* here.
Reported by: gjb
Approved by: bapt (implicit)
MFC after: 1 week
X-MFC-With: r258347
SNI is Server Name Indentification which is a protocol for TLS that
indicates the host that is being connected to at the start of the
handshake. It allows to use Virtual Hosts on HTTPS.
Submitted by: sbz
Submitted by: Michael Gmelin <freebsd@grem.de> [1]
PR: kern/183583 [1]
Reviewed by: des
Approved by: bapt
MFC after: 1 week
request, RFC 2616 14.23 mandates the presence of the Host: header in
all HTTP 1.1 requests.
PR: kern/181445
Submitted by: Kimo <kimor79@yahoo.com>
MFC after: 3 days
the man page) [0]
While here add support for draft-reschke-http-status-308-07
PR: 172451 [0]
Submitted by: gcooper [0]
Reviewed by: des
Approved by: cperciva
MFC after: 1 week
To avoid unexpected process termination from SIGPIPE when writing to a
closed network connection, enable SO_NOSIGPIPE on all network connections.
The POSIX standard MSG_NOSIGNAL is not used since it requires modifying all
send calls to add this flag. This is particularly nasty for SSL connections.
Reviewed by: des
Tested by: bapt
MFC after: 5 days
when there is no timeout, because read(2) will return immediately if there
is no data waiting in the TCP buffer, causing fetch_read() to busy-loop on
slow connections.
MFC after: 3 weeks
Noticed by: Yanhui Shen <shen.elf@gmail.com>
RFC 1738 specifies that any ":", "@", or "/" within a user name or
password in a URL is percent-encoded, to avoid ambiguity with the use
of those characters as URL component separators.
Reviewed by: rstone@
MFC after: 1 month
progress information. The first is that fetch_read() (used in the HTTP
code but not the FTP code) can enter an infinite loop if it has previously
been interrupted by a signal. The second is that when it is interrupted,
fetch_read() will discard any data it may have read up to that point.
Luckily, both bugs are extremely timing-sensitive and therefore difficult
to trigger.
PR: bin/153240
Submitted by: Mark <markjdb@gmail.com>
MFC after: 3 weeks
1. Allow the caller to select active mode.
2. Fix the envar logic so it *always* overrides the caller's flags.
3. Document the change from active to passive.
