Merge pull request #7583 from vector-im/feature/bma/remove_buidkite

Remove usage of Buildkite.
This commit is contained in:
Benoit Marty 2022-11-24 18:51:22 +01:00 committed by GitHub
commit 27419f0d33
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
22 changed files with 342 additions and 53 deletions

View file

@ -13,6 +13,7 @@
* [Code quality](#code-quality)
* [Internal tool](#internal-tool)
* [ktlint](#ktlint)
* [knit](#knit)
* [lint](#lint)
* [Unit tests](#unit-tests)
* [Tests](#tests)
@ -126,6 +127,23 @@ Note that you can run
For ktlint to fix some detected errors for you (you still have to check and commit the fix of course)
#### knit
[knit](https://github.com/Kotlin/kotlinx-knit) is a tool which checks markdown files on the project. Also it generates/updates the table of content (toc) of the markdown files.
So everytime the toc should be updated, just run
<pre>
./gradlew knit
</pre>
and commit the changes.
The CI will check that markdown files are up to date by running
<pre>
./gradlew knitCheck
</pre>
#### lint
<pre>

View file

@ -1,4 +1,4 @@
[![Buildkite](https://badge.buildkite.com/ad0065c1b70f557cd3b1d3d68f9c2154010f83c4d6f71706a9.svg?branch=develop)](https://buildkite.com/matrix-dot-org/element-android/builds?branch=develop)
[![Latest build](https://github.com/vector-im/element-android/actions/workflows/build.yml/badge.svg?query=branch%3Adevelop)](https://github.com/vector-im/element-android/actions/workflows/build.yml?query=branch%3Adevelop)
[![Weblate](https://translate.element.io/widgets/element-android/-/svg-badge.svg)](https://translate.element.io/engage/element-android/?utm_source=widget)
[![Element Android Matrix room #element-android:matrix.org](https://img.shields.io/matrix/element-android:matrix.org.svg?label=%23element-android:matrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#element-android:matrix.org)
[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=vector-im_element-android&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=vector-im_element-android)
@ -14,7 +14,7 @@ It is a total rewrite of [Riot-Android](https://github.com/vector-im/riot-androi
[<img src="resources/img/google-play-badge.png" alt="Get it on Google Play" height="60">](https://play.google.com/store/apps/details?id=im.vector.app)
[<img src="resources/img/f-droid-badge.png" alt="Get it on F-Droid" height="60">](https://f-droid.org/app/im.vector.app)
Nightly build: [![Buildkite](https://badge.buildkite.com/ad0065c1b70f557cd3b1d3d68f9c2154010f83c4d6f71706a9.svg?branch=develop)](https://buildkite.com/matrix-dot-org/element-android/builds?branch=develop) Nightly test status: [![allScreensTest](https://github.com/vector-im/element-android/actions/workflows/nightly.yml/badge.svg)](https://github.com/vector-im/element-android/actions/workflows/nightly.yml)
Build of develop branch: [![GitHub Action](https://github.com/vector-im/element-android/actions/workflows/build.yml/badge.svg?query=branch%3Adevelop)](https://github.com/vector-im/element-android/actions/workflows/build.yml?query=branch%3Adevelop) Nightly test status: [![allScreensTest](https://github.com/vector-im/element-android/actions/workflows/nightly.yml/badge.svg)](https://github.com/vector-im/element-android/actions/workflows/nightly.yml)
# New Android SDK
@ -40,7 +40,7 @@ If you would like to receive releases more quickly (bearing in mind that they ma
1. [Sign up to receive beta releases](https://play.google.com/apps/testing/im.vector.app) via the Google Play Store.
2. Install a [release APK](https://github.com/vector-im/element-android/releases) directly - download the relevant .apk file and allow installing from untrusted sources in your device settings. Note: these releases are the Google Play version, which depend on some Google services. If you prefer to avoid that, try the latest dev builds, and choose the F-Droid version.
3. If you're really brave, install the [very latest dev build](https://buildkite.com/matrix-dot-org/element-android/builds/latest?branch=develop&state=passed) - click on *Assemble (GPlay or FDroid) Debug version* then on *Artifacts*.
3. If you're really brave, install the [very latest dev build](https://github.com/vector-im/element-android/actions/workflows/build.yml?query=branch%3Adevelop) - pick a build, then click on `Summary` to download the APKs from there: `vector-Fdroid-debug` and `vector-Gplay-debug` contains the APK for the desired store. Each file contains 5 APKs. 4 APKs for every supported specific architecture of device. In doubt you can install the `universal` APK.
## Contributing

1
changelog.d/7583.misc Normal file
View file

@ -0,0 +1 @@
Remove usage of Buildkite.

View file

@ -0,0 +1,52 @@
## Installing from CI
<!--- TOC -->
* [Installing from Buildkite](#installing-from-buildkite)
* [Installing from GitHub](#installing-from-github)
* [Create a GitHub token](#create-a-github-token)
* [Provide artifact URL](#provide-artifact-url)
* [Next steps](#next-steps)
* [Future improvement](#future-improvement)
<!--- END -->
Installing APK build by the CI is possible
### Installing from Buildkite
The script `./tools/install/installFromBuildkite.sh` can be used, but Builkite will be removed soon. See next section.
### Installing from GitHub
To install an APK built by a GitHub action, run the script `./tools/install/installFromGitHub.sh`. You will need to pass a GitHub token to do so.
#### Create a GitHub token
You can create a GitHub token going to your Github account, at this page: [https://github.com/settings/tokens](https://github.com/settings/tokens).
You need to create a token (classic) with the scope `repo/public_repo`. So just check the corresponding checkbox.
Validity can be long since the scope of this token is limited. You will still be able to delete the token and generate a new one.
Click on Generate token and save the token locally.
### Provide artifact URL
The script will ask for an artifact URL. You can get this artifact URL by following these steps:
- open the pull request
- in the check at the bottom, click on `APK Build / Build debug APKs`
- click on `Summary`
- scroll to the bottom of the page
- copy the link `vector-Fdroid-debug` if you want the F-Droid variant or `vector-Gplay-debug` if you want the Gplay variant.
The copied link can be provided to the script.
### Next steps
The script will download the artifact, unzip it and install the correct version (regarding arch) on your device.
Files will be added to the folder `./tmp/DebugApks`. Feel free to cleanup this folder from time to time, the script will not delete files.
### Future improvement
The script could ask the user for a Pull Request number and Gplay/Fdroid choice like it was done with Buildkite script. Using GitHub API may be possible to do that.

View file

@ -1,6 +1,6 @@
// Default configuration copied from https://runningcode.github.io/gradle-doctor/configuration/
def isCiBuild = System.env.BUILDKITE == "true" || System.env.GITHUB_ACTIONS == "true"
def isCiBuild = System.env.GITHUB_ACTIONS == "true"
println "Is CI build: $isCiBuild"
doctor {

View file

@ -0,0 +1,95 @@
#!/usr/bin/env bash
# Exit on any error
set -e
if [[ "$#" -ne 1 ]]; then
echo "Usage: $0 GitHub_Token" >&2
echo "Read more about this script in the doc ./docs/installing_from_ci.md"
exit 1
fi
gitHubToken=$1
# Path where the app is cloned (it's where this project has been cloned)
appPath=$(dirname $(dirname $(dirname $0)))
# Path where the APK will be downloaded from CI (it's a dir)
baseImportPath="${appPath}/tmp/DebugApks"
# Select device
serialNumber=$(${appPath}/tools/install/androidSelectDevice.sh)
# Detect device architecture
arch=$(adb -s ${serialNumber} shell getprop ro.product.cpu.abi)
echo
echo "Will install the application on device ${serialNumber} with arch ${arch}"
# Artifact URL
echo
read -p "Artifact url (ex: https://github.com/vector-im/element-android/suites/9293388174/artifacts/435942121)? " artifactUrl
## Example of default value for Gplay
#artifactUrl=${artifactUrl:-https://github.com/vector-im/element-android/suites/9293388174/artifacts/435942121}
## Example of default value for FDroid
# artifactUrl=${artifactUrl:-https://github.com/vector-im/element-android/suites/9293388174/artifacts/435942119}
artifactId=$(echo ${artifactUrl} | rev | cut -d'/' -f1 | rev)
# Download files
targetPath=${baseImportPath}/${artifactId}
filename="artifact.zip"
fullFilePath="${targetPath}/${filename}"
# Check if file already exists
if test -f "$fullFilePath"; then
read -p "$fullFilePath already exists. Override (yes/no) default to no ? " download
download=${download:-no}
else
download="yes"
fi
# Ignore error from now
set +e
if [ ${download} == "yes" ]; then
echo "Downloading ${filename} to ${targetPath}..."
python3 ${appPath}/tools/release/download_github_artifacts.py \
--token ${gitHubToken} \
--artifactUrl ${artifactUrl} \
--directory ${targetPath} \
--filename ${filename} \
--ignoreErrors
fi
echo "Unzipping ${filename}..."
unzip $fullFilePath -d ${targetPath}
## gplay or fdroid
if test -d "${targetPath}/gplay"; then
variant="gplay"
elif test -d "${targetPath}/fdroid"; then
variant="fdroid"
else
echo "No variant found"
exit 1
fi
fullApkPath="${targetPath}/${variant}/debug/vector-${variant}-${arch}-debug.apk"
echo "Installing ${fullApkPath} to device ${serialNumber}..."
adb -s ${serialNumber} install -r ${fullApkPath}
# Check error and propose to uninstall and retry installing
if [[ "$?" -ne 0 ]]; then
read -p "Error, do you want to uninstall the application then retry (yes/no) default to no ? " retry
retry=${retry:-no}
if [ ${retry} == "yes" ]; then
echo "Uninstalling..."
adb -s ${serialNumber} uninstall im.vector.app.debug
echo "Installing again..."
adb -s ${serialNumber} install -r ${fullApkPath}
fi
fi

View file

@ -0,0 +1,154 @@
#!/usr/bin/env python3
#
# Copyright 2022 New Vector Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import argparse
import hashlib
import json
import os
# Run `pip3 install requests` if not installed yet
import requests
# This script downloads artifacts from GitHub.
# Ref: https://docs.github.com/en/rest/actions/artifacts#get-an-artifact
error = False
### Arguments
parser = argparse.ArgumentParser(description='Download artifacts from GitHub.')
parser.add_argument('-t',
'--token',
required=True,
help='The GitHub token with read access.')
parser.add_argument('-a',
'--artifactUrl',
required=True,
help='the artifact_url from GitHub.')
parser.add_argument('-f',
'--filename',
help='the filename, if not provided, will use the artifact name.')
parser.add_argument('-i',
'--ignoreErrors',
help='Ignore errors that can be ignored. Build state and number of artifacts.',
action="store_true")
parser.add_argument('-d',
'--directory',
default="",
help='the target directory, where files will be downloaded. If not provided the build number will be used to create a directory.')
parser.add_argument('-v',
'--verbose',
help="increase output verbosity.",
action="store_true")
parser.add_argument('-s',
'--simulate',
help="simulate action, do not create folder or download any file.",
action="store_true")
args = parser.parse_args()
if args.verbose:
print("Argument:")
print(args)
# Split the artifact URL to get information
# Ex: https://github.com/vector-im/element-android/suites/9293388174/artifacts/435942121
artifactUrl = args.artifactUrl
if not artifactUrl.startswith('https://github.com/'):
print("❌ Invalid parameter --artifactUrl %s. Must start with 'https://github.com/'" % artifactUrl)
exit(1)
if "/artifacts/" not in artifactUrl:
print("❌ Invalid parameter --artifactUrl %s. Must contain '/artifacts/'" % artifactUrl)
exit(1)
artifactItems = artifactUrl.split("/")
if len(artifactItems) != 9:
print("❌ Invalid parameter --artifactUrl %s. Please check the format." % (artifactUrl))
exit(1)
gitHubRepoOwner = artifactItems[3]
gitHubRepo = artifactItems[4]
artifactId = artifactItems[8]
if args.verbose:
print("gitHubRepoOwner: %s, gitHubRepo: %s, artifactId: %s" % (gitHubRepoOwner, gitHubRepo, artifactId))
headers = {
'Authorization': "Bearer %s" % args.token,
'Accept': 'application/vnd.github+json'
}
base_url = "https://api.github.com/repos/%s/%s/actions/artifacts/%s" % (gitHubRepoOwner, gitHubRepo, artifactId)
### Fetch build state
print("Getting artifacts data of project '%s/%s' artifactId '%s'..." % (gitHubRepoOwner, gitHubRepo, artifactId))
if args.verbose:
print("Url: %s" % base_url)
r = requests.get(base_url, headers=headers)
data = json.loads(r.content.decode())
if args.verbose:
print("Json data:")
print(data)
if args.verbose:
print("Create subfolder %s to download artifacts..." % artifactId)
if args.directory == "":
targetDir = artifactId
else:
targetDir = args.directory
if not args.simulate:
os.makedirs(targetDir, exist_ok=True)
url = data.get("archive_download_url")
if args.filename is not None:
filename = args.filename
else:
filename = data.get("name") + ".zip"
## Print some info about the artifact origin
commitLink = "https://github.com/%s/%s/commit/%s" % (gitHubRepoOwner, gitHubRepo, data.get("workflow_run").get("head_sha"))
print("Preparing to download artifact `%s`, built from branch: `%s` (commit %s)" % (data.get("name"), data.get("workflow_run").get("head_branch"), commitLink))
if args.verbose:
print()
print("Artifact url: %s" % url)
target = targetDir + "/" + filename
sizeInBytes = data.get("size_in_bytes")
print("Downloading %s to '%s' (file size is %s bytes, this may take a while)..." % (filename, targetDir, sizeInBytes))
if not args.simulate:
# open file to write in binary mode
with open(target, "wb") as file:
# get request
response = requests.get(url, headers=headers)
# write to file
file.write(response.content)
print("Verifying file size...")
# get the file size
size = os.path.getsize(target)
if sizeInBytes != size:
# error = True
print("Warning, file size mismatch: expecting %s and get %s. This is just a warning for now..." % (sizeInBytes, size))
if error:
print("❌ Error(s) occurred, please check the log")
exit(1)
else:
print("Done!")

View file

@ -214,7 +214,7 @@ else
fi
printf "\n================================================================================\n"
read -p "Wait for Buildkite https://buildkite.com/matrix-dot-org/element-android/builds?branch=main to build the 'main' branch. Press enter when it's done."
read -p "Wait for the GitHub action https://github.com/vector-im/element-android/actions/workflows/build.yml?query=branch%3Amain to build the 'main' branch. Press enter when it's done."
printf "\n================================================================================\n"
printf "Running the release script...\n"

View file

@ -76,15 +76,8 @@ static def gitRevisionDate() {
}
static def gitBranchName() {
def fromEnv = System.env.BUILDKITE_BRANCH as String ?: ""
if (!fromEnv.isEmpty()) {
return fromEnv
} else {
// Note: this command return "HEAD" on Buildkite, so use the system env 'BUILDKITE_BRANCH' content first
def cmd = "git rev-parse --abbrev-ref HEAD"
return cmd.execute().text.trim()
}
def cmd = "git rev-parse --abbrev-ref HEAD"
return cmd.execute().text.trim()
}
// For Google Play build, build on any other branch than main will have a "-dev" suffix
@ -122,8 +115,6 @@ project.android.buildTypes.all { buildType ->
// 64 bits have greater value than 32 bits
ext.abiVersionCodes = ["armeabi-v7a": 1, "arm64-v8a": 2, "x86": 3, "x86_64": 4].withDefault { 0 }
def buildNumber = System.env.BUILDKITE_BUILD_NUMBER as Integer ?: 0
android {
namespace "im.vector.application"
// Due to a bug introduced in Android gradle plugin 3.6.0, we have to specify the ndk version to use
@ -155,7 +146,6 @@ android {
buildConfigField "String", "GIT_REVISION", "\"${gitRevision()}\""
buildConfigField "String", "GIT_REVISION_DATE", "\"${gitRevisionDate()}\""
buildConfigField "String", "GIT_BRANCH_NAME", "\"${gitBranchName()}\""
buildConfigField "String", "BUILD_NUMBER", "\"${buildNumber}\""
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"

View file

@ -1,10 +1,6 @@
## Debug signature
Buildkite CI tool uses docker images to build the Android application, and it looks like the debug signature is changed at each build.
So it's not possible for user to upgrade the application with the last build from buildkite without uninstalling the application.
This folder contains a debug signature, and the debug build will uses this signature to build the APK.
The validity of the signature is 30 years. So it has to be replaced before June 2049 :).

View file

@ -239,7 +239,7 @@ class VectorApplication :
}
private fun logInfo() {
val appVersion = versionProvider.getVersion(longFormat = true, useBuildNumber = true)
val appVersion = versionProvider.getVersion(longFormat = true)
val sdkVersion = Matrix.getSdkVersion()
val date = SimpleDateFormat("MM-dd HH:mm:ss.SSSZ", Locale.US).format(Date())

View file

@ -221,7 +221,6 @@ import javax.inject.Singleton
gitRevision = BuildConfig.GIT_REVISION,
gitRevisionDate = BuildConfig.GIT_REVISION_DATE,
gitBranchName = BuildConfig.GIT_BRANCH_NAME,
buildNumber = BuildConfig.BUILD_NUMBER,
flavorDescription = BuildConfig.FLAVOR_DESCRIPTION,
flavorShortDescription = BuildConfig.SHORT_FLAVOR_DESCRIPTION,
)

View file

@ -117,7 +117,7 @@
android:launchMode="singleTask"
android:windowSoftInputMode="adjustResize" />
<!-- Add tools:ignore="Instantiatable" for the error reported only by Buildkite :/ -->
<!-- Add tools:ignore="Instantiatable" for the error reported only by CI :/ -->
<activity
android:name=".features.media.VectorAttachmentViewerActivity"
android:theme="@style/Theme.Vector.Black.Transparent"
@ -337,7 +337,7 @@
</intent-filter>
</service>
<!-- Add tools:ignore="Instantiatable" for the error reported only by Buildkite and for lintGplayRelease check :/ -->
<!-- Add tools:ignore="Instantiatable" for the error reported only by CI and for lintGplayRelease check :/ -->
<service
android:name=".core.services.VectorSyncAndroidService"
android:exported="false"

View file

@ -24,7 +24,6 @@ data class BuildMeta(
val gitRevision: String,
val gitRevisionDate: String,
val gitBranchName: String,
val buildNumber: String,
val flavorDescription: String,
val flavorShortDescription: String,
)

View file

@ -65,8 +65,7 @@ class LoginSplashFragment :
views.loginSplashVersion.isVisible = true
@SuppressLint("SetTextI18n")
views.loginSplashVersion.text = "Version : ${buildMeta.versionName}\n" +
"Branch: ${buildMeta.gitBranchName}\n" +
"Build: ${buildMeta.buildNumber}"
"Branch: ${buildMeta.gitBranchName} ${buildMeta.gitRevision}"
views.loginSplashVersion.debouncedClicks { navigator.openDebug(requireContext()) }
}
}

View file

@ -94,8 +94,8 @@ class FtueAuthSplashCarouselFragment :
if (buildMeta.isDebug || vectorPreferences.developerMode()) {
views.loginSplashVersion.isVisible = true
@SuppressLint("SetTextI18n")
views.loginSplashVersion.text = "Version : ${buildMeta.versionName}#${buildMeta.buildNumber}\n" +
"Branch: ${buildMeta.gitBranchName}"
views.loginSplashVersion.text = "Version : ${buildMeta.versionName}\n" +
"Branch: ${buildMeta.gitBranchName} ${buildMeta.gitRevision}"
views.loginSplashVersion.debouncedClicks { navigator.openDebug(requireContext()) }
}
views.splashCarousel.registerAutomaticUntilInteractionTransitions()

View file

@ -67,8 +67,7 @@ class FtueAuthSplashFragment :
views.loginSplashVersion.isVisible = true
@SuppressLint("SetTextI18n")
views.loginSplashVersion.text = "Version : ${buildMeta.versionName}\n" +
"Branch: ${buildMeta.gitBranchName}\n" +
"Build: ${buildMeta.buildNumber}"
"Branch: ${buildMeta.gitBranchName} ${buildMeta.gitRevision}"
views.loginSplashVersion.debouncedClicks { navigator.openDebug(requireContext()) }
}
}

View file

@ -283,7 +283,7 @@ class BugReporter @Inject constructor(
.addFormDataPart("user_id", userId)
.addFormDataPart("can_contact", canContact.toString())
.addFormDataPart("device_id", deviceId)
.addFormDataPart("version", versionProvider.getVersion(longFormat = true, useBuildNumber = false))
.addFormDataPart("version", versionProvider.getVersion(longFormat = true))
.addFormDataPart("branch_name", buildMeta.gitBranchName)
.addFormDataPart("matrix_sdk_version", Matrix.getSdkVersion())
.addFormDataPart("olm_version", olmVersion)
@ -305,11 +305,6 @@ class BugReporter @Inject constructor(
}
}
val buildNumber = buildMeta.buildNumber
if (buildNumber.isNotEmpty() && buildNumber != "0") {
builder.addFormDataPart("build_number", buildNumber)
}
// add the gzipped files
for (file in gzippedFiles) {
builder.addFormDataPart("compressed-log", file.name, file.asRequestBody(MimeTypes.OctetStream.toMediaTypeOrNull()))

View file

@ -70,7 +70,7 @@ class VectorUncaughtExceptionHandler @Inject constructor(
val appName = "Element" // TODO Matrix.getApplicationName()
b.append(appName + " Build : " + versionCodeProvider.getVersionCode() + "\n")
b.append("$appName Version : ${versionProvider.getVersion(longFormat = true, useBuildNumber = true)}\n")
b.append("$appName Version : ${versionProvider.getVersion(longFormat = true)}\n")
b.append("SDK Version : ${Matrix.getSdkVersion()}\n")
b.append("Phone : " + Build.MODEL.trim() + " (" + Build.VERSION.INCREMENTAL + " " + Build.VERSION.RELEASE + " " + Build.VERSION.CODENAME + ")\n")

View file

@ -69,10 +69,12 @@ class VectorSettingsHelpAboutFragment :
// application version
findPreference<VectorPreference>(VectorPreferences.SETTINGS_VERSION_PREFERENCE_KEY)!!.let {
it.summary = buildString {
append(versionProvider.getVersion(longFormat = false, useBuildNumber = true))
append(versionProvider.getVersion(longFormat = false))
if (buildMeta.isDebug) {
append(" ")
append(buildMeta.gitBranchName)
append(" ")
append(buildMeta.gitRevision)
}
}

View file

@ -25,7 +25,7 @@ class VersionProvider @Inject constructor(
private val buildMeta: BuildMeta,
) {
fun getVersion(longFormat: Boolean, useBuildNumber: Boolean): String {
fun getVersion(longFormat: Boolean): String {
var result = "${buildMeta.versionName} [${versionCodeProvider.getVersionCode()}]"
var flavor = buildMeta.flavorShortDescription
@ -34,19 +34,10 @@ class VersionProvider @Inject constructor(
flavor += "-"
}
var gitVersion = buildMeta.gitRevision
val gitVersion = buildMeta.gitRevision
val gitRevisionDate = buildMeta.gitRevisionDate
val buildNumber = buildMeta.buildNumber
var useLongFormat = longFormat
if (useBuildNumber && buildNumber != "0") {
// It's a build from CI
gitVersion = "b$buildNumber"
useLongFormat = false
}
result += if (useLongFormat) {
result += if (longFormat) {
" ($flavor$gitVersion-$gitRevisionDate)"
} else {
" ($flavor$gitVersion)"

View file

@ -26,7 +26,6 @@ fun aBuildMeta() = BuildMeta(
gitRevision = "abcdef",
gitRevisionDate = "01-01-01",
gitBranchName = "a-branch-name",
buildNumber = "100",
flavorDescription = "Gplay",
flavorShortDescription = "",
)