Commit graph

7040 commits

Author SHA1 Message Date
Minio Trusted 2174a22835 Update yaml files to latest version RELEASE.2020-07-31T03-39-05Z 2020-07-31 03:58:04 +00:00
poornas a8dd7b3eda
Refactor replication target management. (#10154)
Generalize replication target management so
that remote targets for a bucket can be
managed with ARNs. `mc admin bucket remote`
command will be used to manage targets.
2020-07-30 19:55:22 -07:00
Harshavardhana 25a55bae6f
fix: avoid buffering of server sent events by proxies (#10164) 2020-07-30 19:45:12 -07:00
Harshavardhana fe157166ca
fix: Pass context all the way down to the network call in lockers (#10161)
Context timeout might race on each other when timeouts are lower
i.e when two lock attempts happened very quickly on the same resource
and the servers were yet trying to establish quorum.

This situation can lead to locks held which wouldn't be unlocked
and subsequent lock attempts would fail.

This would require a complete server restart. A potential of this
issue happening is when server is booting up and we are trying
to hold a 'transaction.lock' in quick bursts of timeout.
2020-07-29 23:15:34 -07:00
Adam Brown f7259adf83
Update LastUpdate timestamp before save (#10152) 2020-07-28 13:20:50 -07:00
Harshavardhana 6669560cb9
turn-off bucket usage metrics in gateway mode (#10150)
closes #10147
2020-07-28 13:04:26 -07:00
poornas b46ab7e921
Rename replication target handler (#10142)
Rename replication target handler to a generic bucket target handler
2020-07-28 11:50:47 -07:00
Harshavardhana 27266f8a54
fix: if OPA set do not enforce policy claim (#10149) 2020-07-28 11:47:57 -07:00
poornas 1b6ba0d062
Add validation in cache for offline drives (#10146)
closes #10144
2020-07-28 10:06:52 -07:00
飞雪无情 0d0f09ee66
update and improve zh_CN readme documentation (#10145) 2020-07-28 09:08:22 -07:00
Harshavardhana f200a7fb6a
fix: speed up OBD tests avoid unnecessary memory allocation (#10141)
replace dummy buffer with nullReader{} instead,
to avoid large memory allocations in memory
constrainted environments. allows running
obd tests in such environments.
2020-07-27 14:51:59 -07:00
Minio Trusted 98691f75bc Update yaml files to latest version RELEASE.2020-07-27T18-37-02Z 2020-07-27 18:57:10 +00:00
Harshavardhana 47e304d03c
fix: add missing content-disposition from CORS handler (#10137) 2020-07-27 09:03:38 -07:00
Harshavardhana 9108abf204
fix: allow shareable URLs with rotating creds (#10135)
closes #8935
2020-07-27 09:02:53 -07:00
Harshavardhana 6529dcb3b5
fix: gateway Walk() implementation to list correct contents (#10131)
closes #10122
2020-07-26 22:56:05 -07:00
Harshavardhana 4b22b288a6 add license section be about Apache 2.0
Bonus, remove fossa reference
provides a lot of false positives
and really hard to fix/manage removing
the reference here to avoid any confusion
2020-07-26 01:00:01 -07:00
Harshavardhana abbf6ce6cc
simplify JWKS decoding in OpenID and more tests (#10119)
add tests for non-compliant Azure AD behavior
with "nonce" to fail properly and treat it as
expected behavior for non-standard JWT tokens.
2020-07-25 08:42:41 -07:00
Ritesh H Shukla 416ec316bd
Add clarification for minimum drives per server (#10130) 2020-07-24 22:09:01 -07:00
Harshavardhana 5ffc733eec
fix: enforce bucket quota from browser uploads (#10129) 2020-07-24 21:16:54 -07:00
Minio Trusted 8a23988711 Update yaml files to latest version RELEASE.2020-07-24T22-43-05Z 2020-07-24 23:00:32 +00:00
Harshavardhana 35212b673e
add unformatted disk as part of the error list (#10128)
these errors should be ignored for quorum
error calculation to ensure that we don't
prematurely return unformatted disk error
as part of API calls
2020-07-24 13:16:11 -07:00
Harshavardhana 57ff9abca2
Apply quota usage cache invalidation per second (#10127)
Allow faster lookups for quota check enforcement
2020-07-24 12:24:21 -07:00
Jorge Israel Peña 4752323e1c
Use hdfs.Readdir() to optimize HDFS directory listings (#10121)
Currently, listing directories on HDFS incurs a per-entry remote Stat() call
penalty, the cost of which can really blow up on directories with many
entries (+1,000) especially when considered in addition to peripheral
calls (such as validation) and the fact that minio is an intermediary to the
client (whereas other clients listed below can query HDFS directly).

Because listing directories this way is expensive, the Golang HDFS library
provides the [`Client.Open()`] function which creates a [`FileReader`] that is
able to batch multiple calls together through the [`Readdir()`] function.

This is substantially more efficient for very large directories.

In one case we were witnessing about +20 seconds to list a directory with 1,500
entries, admittedly large, but the Java hdfs ls utility as well as the HDFS
library sample ls utility were much faster.

Hadoop HDFS DFS (4.02s):

    λ ~/code/minio → use-readdir
    » time hdfs dfs -ls /directory/with/1500/entries/
    …
    hdfs dfs -ls   5.81s user 0.49s system 156% cpu 4.020 total

Golang HDFS library (0.47s):

    λ ~/code/hdfs → master
    » time ./hdfs ls -lh /directory/with/1500/entries/
    …
    ./hdfs ls -lh   0.13s user 0.14s system 56% cpu 0.478 total

mc and minio **without** optimization (16.96s):

    λ ~/code/minio → master
    » time mc ls myhdfs/directory/with/1500/entries/
    …
    ./mc ls   0.22s user 0.29s system 3% cpu 16.968 total

mc and minio **with** optimization (0.40s):

    λ ~/code/minio → use-readdir
    » time mc ls myhdfs/directory/with/1500/entries/
    …
    ./mc ls   0.13s user 0.28s system 102% cpu 0.403 total

[`Client.Open()`]: https://godoc.org/github.com/colinmarc/hdfs#Client.Open
[`FileReader`]: https://godoc.org/github.com/colinmarc/hdfs#FileReader
[`Readdir()`]: https://godoc.org/github.com/colinmarc/hdfs#FileReader.Readdir
2020-07-24 11:31:51 -07:00
Klaus Post 11593c6cc4
Usage: Reset merged info when updating (#10126)
When merging multiple buckets reset between each update.

Avoids merging the same usage metrics multiple times resulting 
in duplicate data entries.
2020-07-24 11:02:10 -07:00
Harshavardhana 10025bda45
fix: add missing response headers to CORS handler (#10124) 2020-07-24 00:46:51 -07:00
Praveen raj Mani b800541fbe
fix: a type in NSQ notification target environment key (#10118)
fixes #10100
2020-07-23 12:19:36 -07:00
Harshavardhana 3a73f1ead5
refactor server update behavior (#10107) 2020-07-23 08:03:31 -07:00
Anis Elleuch 1340281cb8
Fix marshaling expiration field in lifecycle (#10117) 2020-07-23 08:01:25 -07:00
poornas b9be841fd2
Add missing validation for replication API conditions (#10114) 2020-07-22 17:39:40 -07:00
Harshavardhana 73890f31af
add minisign verification for container builds (#10115) 2020-07-22 17:09:31 -07:00
Anis Elleuch 456b2ef6eb
Avoid healing to be stuck with many concurrent event listeners (#10111)
If there are many listeners to bucket notifications or to the trace
subsystem, healing fails to work properly since it suspends itself when
the number of concurrent connections is above a certain threshold.

These connections are also continuous and not costly (*no disk access*),
it is okay to just ignore them in waitForLowHTTPReq().
2020-07-22 13:16:55 -07:00
Harshavardhana ad8b53e6d4
add mips64 support for cross compilation (#10106) 2020-07-21 23:56:14 -07:00
Harshavardhana 0b5d1bc91d
fix: bucket replication docs (#10104)
* fix: bucket replication docs

* Update docs/bucket/replication/README.md

Co-authored-by: kannappanr <30541348+kannappanr@users.noreply.github.com>

Co-authored-by: kannappanr <30541348+kannappanr@users.noreply.github.com>
2020-07-21 22:19:30 -07:00
poornas c43da3005a
Add support for server side bucket replication (#9882) 2020-07-21 17:49:56 -07:00
Minio Trusted ca4c15bc63 Update yaml files to latest version RELEASE.2020-07-22T00-26-33Z 2020-07-22 00:44:03 +00:00
Harshavardhana a880283593
Send the lower level error directly from GetDiskID() (#10095)
this is to detect situations of corruption disk
format etc errors quickly and keep the disk online
in such scenarios for requests to fail appropriately.
2020-07-21 13:54:06 -07:00
Bruce Wang e464a5bfbc
Fix bug with fields that contain trimming spaces (#10079)
String x might contain trimming spaces. And it needs to be trimmed. For
example, in csv files, there might be trimming spaces in a field that
ought to meet a query condition that contains the value without
trimming spaces. This applies to both intCast and floatCast functions.
2020-07-21 12:57:09 -07:00
Harshavardhana eb6bf454f1
fix: copyObject encryption from unencrypted object (#10102)
This is a continuation of #10085
2020-07-21 12:25:01 -07:00
Harshavardhana ec06089eda
fix: re-implement cluster healthcheck (#10101) 2020-07-20 18:31:22 -07:00
Harshavardhana 0c4be55936
fix: fix lockup in merge-walk pool (#10098)
Fixes two different types of problems

- continuation of the problem seen in FS #9992
  as not fixed for erasure coded deployments,
  reproduced this issue with spark and its fixed now

- another issue was leaking walk go-routines which
  would lead to high memory usage and crash the system
  this is simply because all the walks which were purged
  at the top limit had leaking end walkers which would
  consume memory endlessly.

closes #9966
closes #10088
2020-07-20 17:28:26 -07:00
Harshavardhana 11d21d5d1b
fix: pass around the correct drives per set (#10097)
this is a precursor change before adding parity
based SLA across zones instead of same stripe size
2020-07-20 16:38:40 -07:00
findmyname666 f9648d3976
add tests lifecycle rules with empty prefix (#10093) 2020-07-20 13:12:50 -07:00
Harshavardhana 2955aae8e4
feat: Add notification support for bucketCreates and removal (#10075) 2020-07-20 12:52:49 -07:00
Harshavardhana 9fd836e51f
add dnsStore interface for upcoming operator webhook (#10077) 2020-07-20 12:28:48 -07:00
Anis Elleuch 518f44908c
fs: Close object fs.json before deletion (#10092)
NFS fails when deleting a file while it is already opened. The reason is
that the object fs.json meta file is opened but not closed before
removal.
2020-07-20 08:52:24 -07:00
Minio Trusted 38f60b3c1d Update yaml files to latest version RELEASE.2020-07-20T02-25-16Z 2020-07-20 02:41:52 +00:00
Harshavardhana e2c71717f8
add different TCP timeouts for internal and incoming (#10090)
closes #10086
2020-07-19 17:16:12 -07:00
Harshavardhana 7764c542f2
allow claims to be optional in STS (#10078)
not all claims need to be present for
the JWT claim, let the policies not
exist and only apply which are present
when generating the credentials

once credentials are generated then
those policies should exist, otherwise
the request will fail.
2020-07-19 15:34:01 -07:00
findmyname666 aa6468932b
make sure lifecycle rule ID is present (#10084) 2020-07-19 15:10:05 -07:00
Harshavardhana 30104cb12b docs: fix veeam document formatting 2020-07-18 18:38:12 -07:00