Commit graph

9597 commits

Author SHA1 Message Date
Harshavardhana c7ed6eee5e
fix: background local test also via channel (#15086)
current implementation for `standalone` setups
was blocking the `perf drive`.

Bonus: remove all old unused complicated code.
2022-06-15 14:51:42 -07:00
Harshavardhana 8082d1fed6
add bucket level S3 received/sent bytes (#15084)
adds bucket level metrics for bytes received and sent bytes on all S3 API calls.
2022-06-14 15:14:24 -07:00
Harshavardhana d2a10dbe69
fix: simplify healthcheck code to freeze calls only once (#15082)
- currently subnet health check was freezing and calling
  locks at multiple locations, avoid them.

- throw errors if first attempt itself fails with no results
2022-06-14 11:22:07 -07:00
Anis Elleuch 14645142db
erasure-sd: Evaluate versioning Prefix in multi-delete objects (#15081)
Erasure SD DeleteObjects() is only inheriting bucket versioning status
from the handler layer.

Add the missing versioning prefix evaluation for each object that will
deleted.
2022-06-14 10:05:12 -07:00
Minio Trusted f34b2ef90b update dashboard Data Usage Growth as time series 2022-06-13 22:05:36 -07:00
George Costea ce894665a8
examples: support configuration of a session policy file (#15078) 2022-06-13 15:36:58 -07:00
Anis Elleuch 0d00f3a55b
kms: initialize after cli parsing (#15076)
KMS depends on the --certs-dir flag. 

Ensure KMS is initialized after loading the flag.
2022-06-13 13:06:13 -07:00
Minio Trusted 48ff373ff7 fix: 'mc support perf drive' crash fix when read returns < 1s 2022-06-13 11:24:37 -07:00
Anis Elleuch e9efee0e64
debug: Close object after check (#15077) 2022-06-13 07:21:04 -07:00
Minio Trusted 4b3e7aee0b Update yaml files to latest version RELEASE.2022-06-11T19-55-32Z 2022-06-11 21:04:23 +00:00
Anis Elleuch dd53b287f2
sts: Avoid printing all STS errors (#15065)
Limit printing STS errors to 

- STS internal error
- STS not initialized
- STS upstream error
2022-06-11 12:55:32 -07:00
Anis Elleuch 21526efe51
Update dperf to 0.4.1 (#15071) 2022-06-11 09:39:50 -07:00
Harshavardhana 7413045f0e
fix: add missing minio_s3_requests_total (#15070)
PR #15052 caused a regression, add the missing metrics back.

Bonus:

- internode information should be only for distributed setups 
- update the dashboard to include 4xx and 5xx error panels.
2022-06-11 00:50:31 -07:00
Harshavardhana d76c508566
debug: verify diff on latest objects on source and target buckets (#15069) 2022-06-10 16:56:51 -07:00
Minio Trusted 8fb46de5e4 Update yaml files to latest version RELEASE.2022-06-10T16-59-15Z 2022-06-10 20:12:04 +00:00
Harshavardhana af1944f28d
support reading systemctl config automatically on baremetal setups (#15066)
this allows for customers to use `mc admin service restart`
directly even when performing RPM, DEB upgrades. Upon such 'restart'
after upgrade MinIO will re-read the /etc/default/minio for any
newer environment variables.

As long as `MINIO_CONFIG_ENV_FILE=/etc/default/minio` is set, this
is honored.
2022-06-10 09:59:15 -07:00
Harshavardhana 214ea14f29
fix: for frozen calls return if client disconnects (#15062) 2022-06-09 05:06:47 -07:00
Anis Elleuch 5fb420c703
prometheus: Add S3 4xx and 5xx S3 monitoring (#15052)
Currently minio_s3_requests_errors_total covers 4xx and 
5xx S3 responses which can be confusing when s3 applications 
sent a lot of HEAD requests with obvious 404 responses or 
when the replication is enabled.

Add 
- minio_s3_requests_4xx_errors_total
- minio_s3_requests_5xx_errors_total

to help users monitor 4xx and 5xx HTTP status codes separately.
2022-06-08 11:22:34 -07:00
Harshavardhana 2420f6c000
fix: make metrics endpoint responsive by reducing the chatter (#15055)
peerOnlineCounter was making NxN calls to many peers, this
can be really long and tedious if there are random servers
that are going down.

Instead we should calculate online peers from the point of
view of "self" and return those online and offline appropriately
by performing a healthcheck.
2022-06-08 02:43:13 -07:00
Harshavardhana b0d7332a0c
healthcheck cluster endpoint should honor write/readQuorum per pool (#15053) 2022-06-07 19:08:21 -07:00
Daniel Valdivia f71b56a5d0
Bump Console v0.18.1 (#15051)
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2022-06-07 12:19:38 -07:00
Harshavardhana d55efc791f
relax O_DIRECT in single drive mode if unsupported (#15045) 2022-06-07 06:44:01 -07:00
Minio Trusted f63645546d update minimum goroutine threshold on dashboard 2022-06-06 22:13:54 -07:00
Kaan Kabalak e2dd3e3587
Include the entirety of vendor folder in .gitignore (#15046)
The 'go mod vendor' command generates a directory called 
'vendor' in the main module's root directory, which includes 
the required packages to support builds. Therefore, we can 
include the 'vendor' directory in .gitignore completely, 
regardless of any file extension.
2022-06-06 20:47:51 -07:00
Minio Trusted 27ab780317 Update yaml files to latest version RELEASE.2022-06-07T00-33-41Z 2022-06-07 01:06:59 +00:00
Minio Trusted e2d4d097e7 do not print errors upon 'nil' err 2022-06-06 17:33:41 -07:00
Minio Trusted ac8cb6ba0d Update yaml files to latest version RELEASE.2022-06-06T23-14-52Z 2022-06-06 23:47:31 +00:00
Shireesh Anjal 4ce81fd07f
Add periodic callhome functionality (#14918)
* Add periodic callhome functionality

Periodically (every 24hrs by default), fetch callhome information and
upload it to SUBNET.

New config keys under the `callhome` subsystem:

enable - Set to `on` for enabling callhome. Default `off`
frequency - Interval between callhome cycles. Default `24h`

* Improvements based on review comments

- Update `enableCallhome` safely
- Rename pctx to ctx
- Block during execution of callhome
- Store parsed proxy URL in global subnet config
- Store callhome URL(s) in constants
- Use existing global transport
- Pass auth token to subnetPostReq
- Use `config.EnableOn` instead of `"on"`

* Use atomic package instead of lock

* Use uber atomic package

* Use `Cancel` instead of `cancel`

Co-authored-by: Harshavardhana <harsha@minio.io>

Co-authored-by: Harshavardhana <harsha@minio.io>
Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
2022-06-06 16:14:52 -07:00
Harshavardhana df9eeb7f8f
fix: do not log concurrently when multiple disks return errors (#15044)
since the values inside 'context' are mutated internally by
logger, make sure to log serially upon errors not concurrently.
2022-06-06 15:15:11 -07:00
Harshavardhana 31c4fdbf79
fix: resyncing 'null' version on pre-existing content (#15043)
PR #15041 fixed replicating 'null' version however
due to a regression from #14994 caused the target
versions for these 'null' versioned objects to have
different 'versions', this may cause confusion with
bi-directional replication and cause double replication.

This PR fixes this properly by making sure we replicate
the correct versions on the objects.
2022-06-06 15:14:56 -07:00
Harshavardhana 48e367ff7d
reject resync start on misconfigured replication rules (#15041)
we expect resync to start on buckets with replication
rule ExistingObjects enabled, if not we reject such
calls.
2022-06-06 02:54:39 -07:00
Anis Elleuch fd02492cb7
avoid limits on the number of parallel trace/bucket notifications listeners (#14799)
Simplifies overall limits on the incoming listeners for notifications.

Fixes #14566
2022-06-05 14:29:12 -07:00
Aditya Manthramurthy addfa35d93
Add FIPS build to CI and add README.fips.md (#15038) 2022-06-04 18:25:37 -07:00
Harshavardhana 5afdc56796
allow single drive mode to run on root disk (#15037)
for practical reasons, allow root disk based installs for single drive mode.
2022-06-03 12:53:42 -07:00
Harshavardhana fb1c333a83 update latest dperf v0.4.0 2022-06-03 11:13:20 -07:00
Harshavardhana c3e1da8e04
honor canceled context and do not leak on mergeChannels (#15034)
mergeEntryChannels has the potential to perpetually
wait on the results channel, context might be closed
and we did not honor the caller context canceling.
2022-06-03 05:59:02 -07:00
Anis Elleuch 20a753e2e5
Fix a possible service freeze after perf object (#15036)
The S3 service can be frozen indefinitely if a client or mc asks for object
perf API but quits early or has some networking issues. The reason is
that partialWrite() can block indefinitely.

This commit makes partialWrite() listens to context cancellation as well. It
also renames deadlinedCtx to healthCtx since it covers handler context
cancellation and not only not only the speedtest deadline.
2022-06-03 05:58:45 -07:00
Minio Trusted 3a398775fb Update yaml files to latest version RELEASE.2022-06-03T01-40-53Z 2022-06-03 02:36:56 +00:00
Aditya Manthramurthy 61a7434379
Update --version option behavior (#15032)
- Add git commit ID
- Add go version
2022-06-02 18:40:53 -07:00
Aditya Manthramurthy 09f5e29327
Bump up console to v0.18.0 (#15031) 2022-06-02 17:34:37 -07:00
Poorna 29edb4ccfe
fix: site replication bucket heal to not panic if replication config is missing (#15025) 2022-06-02 12:34:03 -07:00
Minio Trusted 197d6fb644 Update yaml files to latest version RELEASE.2022-06-02T16-16-26Z 2022-06-02 17:46:32 +00:00
Anis Elleuch d4e565e595
Add defensive check for one stream message size (#15029)
In a streaming response, the client knows the size of a streamed
message but never checks the message size. Add the check to error 
out if the response message is truncated.
2022-06-02 09:16:26 -07:00
Minio Trusted 1fce2b180f Update yaml files to latest version RELEASE.2022-06-02T02-11-04Z 2022-06-02 02:42:14 +00:00
Aditya Manthramurthy be6ccd129d
fix: typo in FIPS sha256 (#15024) 2022-06-01 19:11:04 -07:00
Klaus Post f7cecf0945
Make isIndexedMetaV2 return errors (#15012)
Indexed streams would be decoded by the legacy loader if there 
was an error loading it. Return an error when the stream is indexed 
and it cannot be loaded.

Fixes "unknown minor metadata version" on corrupted xl.meta files and 
returns an actual error.
2022-05-31 19:06:57 -07:00
Harshavardhana 7b2198f7e5
handle IPv6 sourceIPs properly (#15005) 2022-05-31 06:04:12 -07:00
Harshavardhana 52221db7ef
fix: for unexpected errors in reading versioning config panic (#14994)
We need to make sure if we cannot read bucket metadata
for some reason, and bucket metadata is not missing and
returning corrupted information we should panic such
handlers to disallow I/O to protect the overall state
on the system.

In-case of such corruption we have a mechanism now
to force recreate the metadata on the bucket, using
`x-minio-force-create` header with `PUT /bucket` API
call.

Additionally fix the versioning config updated state
to be set properly for the site replication healing
to trigger correctly.
2022-05-31 02:57:57 -07:00
Harshavardhana befbf48563 fix: s3-check-md5 to not panic for incomplete md5 2022-05-30 20:58:42 -07:00
Anis Elleuch 56a61bab56
test: Add GetObjectNInfo test with some outdated disks (#15004)
Add a test reading an object which has some old data in some outdated
disks, in a versioned and non-versioned bucket.
2022-05-30 17:52:59 -07:00