self-hosted/minio
self-hosted/minio
1
0
Fork 0
mirror of https://github.com/minio/minio synced 2024-07-22 02:44:15 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
11994 commits 5 branches 490 tags 174 MiB
Commit graph

6142 commits

Author SHA1 Message Date
Harshavardhana 2dd8faaedc remove unnecessary log in Listing() 2024-06-07 14:52:55 -07:00
Krishnan Parthasarathi 069c4015cd
Don't tier directory objects (#19891) 
Directory objects are used by applications that simulate the folder
structure of an on-disk filesystem. These are zero-byte objects with names
ending with '/'. They are only used to check whether a 'folder' exists in
the namespace.
 2024-06-07 08:43:17 -07:00
Shubhendu 2f6e03fb60
Calculate correct object size while replication (#19888) 
It was missing in case of `replicateObject` but was present for
`replicateAll` already

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-06-06 12:31:01 -07:00
Klaus Post 0fbb945e13
Disable caching of encrypted objects (#19890) 
Don't write encrypted objects to cache, if configured.
 2024-06-06 11:39:18 -07:00
Anis Eleuch b94dd835c9
decom: Fix CurrentSize output when generating the status (#19883) 
StartSize starts with the raw free space of all disks in the given pool,
however during the status, CurrentSize is not showing the current free
raw space, as expected at least by `mc admin decom status` since it was
written.
 2024-06-06 07:30:43 -07:00
Poorna 5aaef9790f
replication: pass checksum headers to replica (#19834) 2024-06-06 02:36:42 -07:00
Bala FA 7edc352d23
Add ILM metrics in metrics-v3 (#19539) 
Signed-off-by: Bala.FA <bala@minio.io>
 2024-06-06 02:36:25 -07:00
Poorna 850a84b08a
simplify site replication multipart proxying (#19885) 2024-06-05 18:01:15 -07:00
Taran Pelkey 4148754ce0
Check both given and normalized group DN on LDAP policy detach requests (#19876) 2024-06-05 15:42:40 -07:00
Harshavardhana 2107722829
upgrade go-oidc to fix GO-2024-2631 (#19884) 2024-06-05 15:00:34 -07:00
jiuker d326ba52e9
feat: support batchJob for windows (#19877) 2024-06-05 08:44:53 -07:00
Sveinn 91e1487de4
Add LDAP public key authentication to SFTP (#19833) 2024-06-05 00:51:13 -07:00
jiuker 90a9f2dd70
fix: log diskerror when detect the disk space failed (#19861) 2024-06-04 09:42:03 -07:00
Harshavardhana d5e48cfd65
fix: remove DriveOPTimeout for REST callers as they don't work properly (#19873) 
Go's net/http is notoriously difficult to have a streaming
deadlines per READ/WRITE on the net.Conn if we add them they
interfere with the Go's internal requirements for a HTTP
connection.

Remove this support for now

fixes #19853
 2024-06-04 08:12:57 -07:00
Anis Eleuch d274566463
race: Fix rare race detected by testing (#19872) 
Below is the race warning:

```
WARNING: DATA RACE
Write at 0x00c02d3d27c0 by goroutine 1210:
  github.com/minio/minio/cmd.(*healingTracker).bucketDone()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:273 +0x13a
  github.com/minio/minio/cmd.(*erasureObjects).healErasureSet()
      github.com/minio/minio/cmd/global-heal.go:525 +0x2158
  github.com/minio/minio/cmd.healFreshDisk()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:450 +0x107e
  github.com/minio/minio/cmd.monitorLocalDisksAndHeal.func1()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:528 +0x150
  github.com/minio/minio/cmd.monitorLocalDisksAndHeal.gowrap2()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:538 +0x82

Previous read at 0x00c02d3d27c0 by goroutine 1446:
  github.com/minio/minio/cmd.(*erasureObjects).healErasureSet.func5()
      github.com/minio/minio/cmd/global-heal.go:232 +0xfd
```
 2024-06-04 08:12:32 -07:00
Shubhendu 21b6204692
Test proxying of DEL marker for bucket replication (#19870) 
Make sure to avoid proxying for DEL markers

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-06-04 04:38:26 -07:00
Taran Pelkey d98faeb26a
Check if LDAP User has attached policy before creating Service Account (#19843) 
Check if ldap user has policy before creating
 2024-06-03 12:58:48 -07:00
Klaus Post 0a63dc199c
Add trace sizes to more trace types (#19864) 
Add trace sizes to

* ILM traces
* Replication traces
* Healing traces
* Decommission traces
* Rebalance traces
* (s)ftp traces
* http traces.
 2024-06-03 08:45:54 -07:00
Klaus Post e72429c79c
Add sizes to traces (#19851) 
added to storage and grid traces. Can provide more context for traces that aren't HTTP. Others may apply.
 2024-05-31 22:17:37 -07:00
Klaus Post c5b3f5553f
Add per connection RPC metrics (#19852) 
Provides individual and aggregate stats for each RPC connection.

Example:

```
  "rpc": {
   "collectedAt": "2024-05-31T14:33:29.1373103+02:00",
   "connected": 30,
   "disconnected": 0,
   "outgoingStreams": 69,
   "incomingStreams": 0,
   "outgoingBytes": 174822796,
   "incomingBytes": 175821566,
   "outgoingMessages": 768595,
   "incomingMessages": 768589,
   "outQueue": 0,
   "lastPongTime": "2024-05-31T12:33:28Z",
   "byDestination": {
    "http://127.0.0.1:9001": {
     "collectedAt": "2024-05-31T14:33:29.1373103+02:00",
     "connected": 5,
     "disconnected": 0,
     "outgoingStreams": 2,
     "incomingStreams": 0,
     "outgoingBytes": 38432543,
     "incomingBytes": 66604052,
     "outgoingMessages": 229496,
     "incomingMessages": 229575,
     "outQueue": 0,
     "lastPongTime": "2024-05-31T12:33:27Z"
    },
    "http://127.0.0.1:9002": {
     "collectedAt": "2024-05-31T14:33:29.1373103+02:00",
     "connected": 5,
     "disconnected": 0,
     "outgoingStreams": 6,
     "incomingStreams": 0,
     "outgoingBytes": 38215680,
     "incomingBytes": 66121283,
     "outgoingMessages": 228525,
     "incomingMessages": 228510,
     "outQueue": 0,
     "lastPongTime": "2024-05-31T12:33:27Z"
    },
...
```
 2024-05-31 22:16:24 -07:00
Klaus Post d3ae0aaad3
Add max buffering to SFTP (#19848) 
Prevent OOM by adversarial use of SFTP upload by setting a 100MB max upload buffer.
 2024-05-31 14:28:07 -07:00
Anis Eleuch 1277ad69a6
heal: Remove .healing.bin when all ES drives are healing (#19846) 
In the very rare case when all drives in a erasure set need to be healed,
remove .healing.bin from all drives, otherwise it will be stuck in a
loop

Also, fix a unit test that fails sometimes due to wrong test.
 2024-05-31 07:48:50 -07:00
Harshavardhana 8f93e81afb
change service account embedded policy size limit (#19840) 
Bonus: trim-off all the unnecessary spaces to allow
for real 2048 characters in policies for STS handlers
and re-use the code in all STS handlers.
 2024-05-30 11:10:41 -07:00
Harshavardhana 4af31e654b
avoid pre-populating buffers for deployments < 32GiB memory (#19839) 2024-05-30 04:58:12 -07:00
Harshavardhana aad50579ba
fix: wire up ILM sub-system properly for help (#19836) 2024-05-30 01:14:58 -07:00
Harshavardhana 38d059b0ae
fix: single node multi-drive must register local drives properly (#19832) 
since #19688 there was a regression introduced during drive
lookups for single node multi-drive setups, drive replacement
would not work correctly without this PR.
 2024-05-29 13:12:44 -07:00
Klaus Post bd4eeb4522
Fix flipped EcM, EcN in metadata header (#19831) 
Since this is a tuple encoded field we can just flip the struct members.
 2024-05-29 12:14:09 -07:00
jiuker 03e3493288
fix: correct parse the tagging error for PostPolicyBucketHandler (#19825) 2024-05-29 11:50:46 -07:00
Harshavardhana 64baedf5a4
fix: hide prefixes for Hadoop properly (#19821) 2024-05-28 15:53:15 -07:00
Anis Eleuch f79a4ef4d0
policy: More defensive code validating svc:DurationSeconds (#19820) 
This does not fix any current issue, but merging https://github.com/minio/madmin-go/pull/282
can lose the validation of the service account expiration time.

Add more defensive code for now. In the future, we should avoid doing
validation in another library.
 2024-05-28 10:19:04 -07:00
Taran Pelkey 2d53854b19
Restrict access keys for users and groups to not allow '=' or ',' (#19749) 
* initial commit

* Add UTF check

---------

Co-authored-by: Harshavardhana <harsha@minio.io>
 2024-05-28 10:14:16 -07:00
jiuker c904ef966e
feat: support tags for PostPolicy upload (#19816) 2024-05-27 21:44:00 -07:00
Harshavardhana e0fe7cc391
fix: information disclosure bug in preconditions GET (#19810) 
precondition check was being honored before, validating
if anonymous access is allowed on the metadata of an
object, leading to metadata disclosure of the following
headers.

```
Last-Modified
Etag
x-amz-version-id
Expires:
Cache-Control:
```

although the information presented is minimal in nature,
and of opaque nature. It still simply discloses that an
object by a specific name exists or not without even having
enough permissions.
 2024-05-27 12:17:46 -07:00
Harshavardhana 9d20dec56a Revert "remove dataErrs from er.deleteIfDangling code" 
This reverts commit 7d75b1e758.

This fails multipart tests we need this code to handle
existing challenges, so wait for the comprehensive fix.
 2024-05-26 11:13:29 -07:00
Harshavardhana 597a785253
fix: authenticate LDAP via actual DN instead of normalized DN (#19805) 
fix: authenticate LDAP via actual DN instead of normalized DN

Normalized DN is only for internal representation, not for
external communication, any communication to LDAP must be
based on actual user DN. LDAP servers do not understand
normalized DN.

fixes #19757
 2024-05-25 06:43:06 -07:00
Harshavardhana 7d75b1e758 remove dataErrs from er.deleteIfDangling code 
avoid this until a comprehensive change is
merged such as https://github.com/minio/minio/pull/19797
 2024-05-24 18:20:04 -07:00
Aditya Manthramurthy 5f78691fcf
ldap: Add user DN attributes list config param (#19758) 
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.

A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.

This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
 2024-05-24 16:05:23 -07:00
Shireesh Anjal a591e06ae5
Add cluster scanner metrics in metrics-v3 (#19517) 
endpoint: /minio/metrics/v3/cluster/scanner
metrics:
 - bucket_scans_finished (counter)
 - bucket_scans_started (counter)
 - directories_scanned (counter)
 - last_activity_nano_seconds (gauge)
 - objects_scanned (counter)
 - versions_scanned (counter)
 2024-05-24 12:29:25 -07:00
Harshavardhana 443c93c634
compute time spent in ILM properly (#19806) 2024-05-24 12:28:51 -07:00
Shireesh Anjal 5659cddc84
Add cluster config metrics in metrics-v3 (#19507) 
endpoint: /minio/metrics/v3/cluster/config
metrics:
- write_quorum
- rrs_parity
- standard_parity
 2024-05-24 05:50:46 -07:00
Shireesh Anjal 673a521711
Change endpoint of v3 notification metrics (#19804) 
from /cluster/notification to /notification
 2024-05-24 04:10:24 -07:00
Shireesh Anjal 7981509cc8
Add cluster and bucket replication metrics in metrics-v3 (#19546) 
endpoint: /minio/metrics/v3/cluster/replication
metrics:
- average_active_workers
- average_queued_bytes
- average_queued_count
- average_transfer_rate
- current_active_workers
- current_transfer_rate
- last_minute_queued_bytes
- last_minute_queued_count
- max_active_workers
- max_queued_bytes
- max_queued_count
- max_transfer_rate
- recent_backlog_count

endpoint: /minio/metrics/v3/api/bucket/replication
metrics:
- last_hour_failed_bytes
- last_hour_failed_count
- last_minute_failed_bytes
- last_minute_failed_count
- latency_ms
- proxied_delete_tagging_requests_total
- proxied_get_requests_failures
- proxied_get_requests_total
- proxied_get_tagging_requests_failures
- proxied_get_tagging_requests_total
- proxied_head_requests_failures
- proxied_head_requests_total
- proxied_put_tagging_requests_failures
- proxied_put_tagging_requests_total
- sent_bytes
- sent_count
- total_failed_bytes
- total_failed_count
- proxied_delete_tagging_requests_failures
 2024-05-23 00:41:18 -07:00
Harshavardhana d38e020b29
remove errant logs for disconnected remote (#19793) 
Signed-off-by: Harshavardhana <harsha@minio.io>
 2024-05-22 18:12:23 -07:00
Poorna 7d29030292
fix list results returned for spark max-keys=2 listing (#19791) 
This PR continues fix #19725 for some unhandled cases
 2024-05-22 16:16:34 -07:00
Shubhendu 7c7650b7c3
Add sufficient deadlines and countermeasures to handle hung node scenario (#19688) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Harshavardhana <harsha@minio.io>
 2024-05-22 16:07:14 -07:00
Harshavardhana ca80eced24
usage of deadline conn at Accept() breaks websocket (#19789) 
fortunately not wired up to use, however if anyone
enables deadlines for conn then sporadically MinIO
startups fail.
 2024-05-22 10:49:27 -07:00
jiuker 9906b3ade9
fix: reject ilm rule when bucket LockEnabled (#19785) 2024-05-21 23:50:03 -07:00
Anis Eleuch bf1769d3e0
xl: Avoid marking a drive offline after one part read failure (#19779) 
This commit will fix one rare case of a multipart object that
can be read in theory but GetObject API returned an error.

It turned out that a six years old code was marking a drive offline
when the bitrot streaming fails to read a part in a disk with any error.
This can affect reading a subsequent part, though having enough shards,
but unable to construct because one drive was marked offline earlier.

This commit will remove the drive marking offline code. It will also
close the bitrotstreaming reader before marking it as nil.
 2024-05-21 07:36:21 -07:00
Harshavardhana 63e1ad9f29 fix: the user-agent for Veeam 2024-05-20 11:54:52 -07:00
Harshavardhana 1fd90c93ff
re-use StorageAPI while loading drive formats (#19770) 
Bonus: safe settings for deployment ID to avoid races
 2024-05-19 01:06:49 -07:00