This can happen if a test is approved as failing, it then is fixed, and a
changelist needs to break it again, and the new failure is pre-approved.
This causes approve_results to warn that it can't properly change an approval
from one outcome to the exact same outcome, which is of course wrong, it can
do that.
Change-Id: Ib9ea1d4ea5a204b66808a1789c31fe6ec9c2c9d2
Reviewed-on: https://dart-review.googlesource.com/c/86920
Reviewed-by: William Hesse <whesse@google.com>
This change ads two new fields to approved_results.json:
* approver: The username of the person who approved the new results.
* approved_at: The ISO 8601 UTC time of when the approval occured.
This data provides a history for approvals as a replacement for the git
commit history for status files. Each build makes a copy of the
approved_results.json file at the time, effectively acting as a commit.
Bug: https://github.com/dart-lang/sdk/issues/35315
Change-Id: I64430a53820a0cffeb277012d5318b50999a9df5
Reviewed-on: https://dart-review.googlesource.com/c/85940
Reviewed-by: William Hesse <whesse@google.com>
This change lets committers preapprove new failing tests based on commit
queue / dry run results for CQ builders, such that the CQ builder stays
green when the change is submitted.
The current implementation does not support changing one test from a failing
mode to another failing mode. It will work, but if the corresponding CI
builder cycles before the changelist is submitted, it will become red until
the changelist is submitted. This will be addressed in the future.
Likewise approving a builder after a preapproval will undo the preapproval
unless the changelist has been submitted and the corresponding CI builder
has cycled.
Bug: https://github.com/dart-lang/sdk/issues/35289
Change-Id: Ie43df4439a27a3472c5178cb7c6af69292cf70e4
Reviewed-on: https://dart-review.googlesource.com/c/85601
Reviewed-by: William Hesse <whesse@google.com>
test.dart locates where the current branch branched off master and compares
the local testing results with the appropriate mainline builder results,
letting you know how the current change compares without the need for status
files.
Bug: https://github.com/dart-lang/sdk/issues/35086
Change-Id: Ib79479b867c5ac131302fea1bdf7effd0422a83a
Reviewed-on: https://dart-review.googlesource.com/c/83281
Reviewed-by: Alexander Thomas <athom@google.com>
Change-Id: Ia20e682c542edba8b606276124e9f8fd78785680
Reviewed-on: https://dart-review.googlesource.com/c/83620
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
The approve_results program downloads the current results, the approved
results, and flakiness information for the specified bots. It presents
tables of the tests that have been fixed and the ones that have broken since
the last approval. Finally, it provides the option to approve the new
changes.
The approve_results program is part of the new workflow and only affects the
greenness of bots enrolled in it.
You can list the available bots by running:
dart tools/approve_results.dart -l
You can approve the results on a bot by running:
dart tools/approve_results.dart -b vm-kernel-linux-release-x64
Globs are supported, e.g. to approve all vm linux bots on ia32 and x64:
dart tools/approve_results.dart -b 'vm-*linux*{ia32,x64}*'
By default the tool is interactive and asks whether the changes should be
approved. Approving the results turns the bots green on the next commit.
It's assumed that bugs have been filed prior to approving the test changes.
The -y option approves the changes non-interactively while the -n option
does a try run and shows what needs to be approved.
This tool is meant to be used early and often to turn the bot green. Emails
are not set about additional failures until the regressions has been
approved.
Bug: https://github.com/dart-lang/sdk/issues/35000
Change-Id: I6479edaa76556aa5ef499509c04b92ae770d1727
Reviewed-on: https://dart-review.googlesource.com/c/82222
Reviewed-by: William Hesse <whesse@google.com>
