2016-01-26 20:23:21 +00:00
# Contributing to Tokei
2020-05-09 17:46:13 +00:00
- [Language Addition ](#language-addition )
- [Bug Reports ](#bug-reports )
2016-01-26 20:23:21 +00:00
# Language Addition
2020-05-09 17:46:13 +00:00
2022-08-14 15:22:26 +00:00
Currently, Tokei generates languages from the [`languages.json` ](languages.json )
file. JSON was chosen to make it easy to add new languages and change code
structure without changing large data structures. Here, we will go over the
properties of a language in `languages.json` through examples.
2016-12-03 11:51:34 +00:00
2020-03-18 10:21:39 +00:00
```json
"JavaScript": {
"line_comment": ["//"],
"multi_line_comments": [["/*", "*/"]],
"quotes": [["\\\"", "\\\""]],
"quotes": [["\\\"", "\\\""], ["'", "'"], ["`", "`"]],
"extensions": ["js", "mjs"]
2016-12-03 11:51:34 +00:00
},
```
Above is the JavaScript's definition. The first thing that needs to be defined
2022-08-14 15:22:26 +00:00
is the key. The key's format should be same as [Rust's enum style]. As this key
will be used in an enum for identifying the language. For a lot of languages,
2020-05-09 17:46:13 +00:00
this also works for showing the language when we print to the screen.
2020-02-14 10:43:40 +00:00
2022-08-14 15:22:26 +00:00
However, there are some languages whose names don't work with the enum style.
For example, `JSON` is usually shown in all caps, but that doesn't fit in Rust's
enum style. So we have an additional optional field called `name` which defines
2016-12-03 11:51:34 +00:00
how the language should look when displayed to the user.
```json
2022-08-14 15:22:26 +00:00
"Json": {
2016-12-03 11:51:34 +00:00
"name": "JSON",
2022-08-14 15:22:26 +00:00
//...
},
2016-12-03 11:51:34 +00:00
```
2022-08-14 15:22:26 +00:00
For defining comments, there are a few properties. The most commonly used
property is `line_comment` which defines single line comments. These are comments
which don't continue onto the next line. Here is an example in Rust:
2016-12-03 11:51:34 +00:00
```rust
let x = 5; // default x position
let y = 0; // default y position
```
2020-05-09 17:46:37 +00:00
The `line_comment` property expects an array of strings, as some languages have
2022-08-14 15:22:26 +00:00
multiple syntaxes for defining a single line comment. For example, `PHP` allows
both `#` and `//` for single line comments.
2016-12-03 11:51:34 +00:00
```json
"Php": {
2020-02-20 10:19:12 +00:00
"line_comment": [
2016-12-03 11:51:34 +00:00
"#",
"//"
2022-08-14 15:22:26 +00:00
],
//...
},
2016-12-03 11:51:34 +00:00
```
2022-08-14 15:22:26 +00:00
For defining comments that also have an ending syntax, there is the `multi_line`
property. An example for such comments in Rust:
2016-12-03 11:51:34 +00:00
2018-04-08 10:55:27 +00:00
```rust
2016-12-03 11:51:34 +00:00
let x = /* There is a reason
2022-08-14 15:22:26 +00:00
for this comment, I swear! */
2016-12-03 11:51:34 +00:00
10;
```
2020-05-28 05:55:14 +00:00
The `verbatim_quotes` property expects an array of strings, as some languages
have multiple syntaxes for defining verbatim strings. A verbatim string
in the context of Tokei is a string literal that can have unescaped `"` s. For example [`CSharp` ](https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/strings/#regular-and-verbatim-string-literals )
```json
"CSharp": {
"verbatim_quotes": [
[
"@\\\"",
"\\\""
]
2022-08-14 15:22:26 +00:00
],
//...
},
2020-05-28 05:55:14 +00:00
```
```csharp
const string BasePath = @"C:\";
```
2018-04-08 10:55:27 +00:00
Some languages have a single, standard filename with no extension
like `Makefile` or `Dockerfile` . These can be defined with the
`filenames` property:
2017-06-05 14:28:52 +00:00
```json
2022-08-14 15:22:26 +00:00
"Makefile": {
"filenames": [
2017-06-05 14:28:52 +00:00
"makefile"
],
2022-08-14 15:22:26 +00:00
"extensions": [
2017-06-05 14:28:52 +00:00
"makefile",
"mak",
"mk"
]
2022-08-14 15:22:26 +00:00
},
2017-06-05 14:28:52 +00:00
```
2018-04-08 10:55:27 +00:00
Filenames should be all-lowercase, whether or not the filename
typically has capital letters included.
2017-06-05 14:28:52 +00:00
2022-08-14 15:22:26 +00:00
Note that filenames will **override** extensions. With the
following definition, a file named `CMakeLists.txt` will be
2018-04-08 10:55:27 +00:00
detected as a `CMake` file, not a `Text` file.
2017-06-05 14:28:52 +00:00
```json
2022-08-14 15:22:26 +00:00
"Text": {
"extensions": [
2017-06-05 14:28:52 +00:00
"txt"
]
},
2022-08-14 15:22:26 +00:00
"CMake": {
2017-06-05 14:28:52 +00:00
"filenames": [
"cmakelists.txt"
]
2022-08-14 15:22:26 +00:00
},
2018-04-08 10:55:27 +00:00
```
# Tests
2020-05-09 17:46:13 +00:00
2022-08-14 15:22:26 +00:00
A test file is required for language additions. The file should
2018-04-08 10:55:27 +00:00
contain every variant comments and quotes, as well as a comment
at the top of the file containing the manually verified lines,
2022-08-14 15:22:26 +00:00
code, comments, blanks in the following format:
2020-02-14 10:43:40 +00:00
2022-08-14 15:22:26 +00:00
```
2020-02-14 10:43:40 +00:00
NUM lines NUM code NUM comments NUM blanks
```
### Example
2020-05-09 17:46:13 +00:00
2022-08-14 15:22:26 +00:00
In Rust for example, the first line should look like the following:
2020-06-19 06:44:03 +00:00
```rust
//! 39 lines 32 code 2 comments 5 blanks
2020-02-14 10:43:40 +00:00
```
The comment should use the syntax of the language you're testing.
2022-08-14 15:22:26 +00:00
A good example of a test file is [`tests/data/rust.rs` ](tests/data/rust.rs ).
2018-04-08 10:55:27 +00:00
```rust
2022-08-14 15:22:26 +00:00
//! 48 lines 36 code 6 comments 6 blanks
//! ```rust
//! fn main () {
//! // Comment
//!
//! println!("Hello World!");
//! }
//! ```
2018-04-08 10:55:27 +00:00
/* /**/ */
fn main() {
2022-08-14 15:22:26 +00:00
let start = r##"/*##\"
2020-05-28 05:55:14 +00:00
\"##;
// comment
2018-04-08 10:55:27 +00:00
loop {
if x.len() >= 2 & & x[0] == '*' & & x[1] == '/' { // found the */
break;
}
}
}
2017-06-05 14:28:52 +00:00
2020-05-09 17:46:13 +00:00
fn foo< 'a, 'b>(name: & 'b str) {
2018-04-08 10:55:27 +00:00
let this_ends = "a \"test/*.";
call1();
call2();
let this_does_not = /* a /* nested */ comment " * /
"*/another /*test
call3();
*/";
}
fn foobar() {
let does_not_start = // "
"until here,
test/*
test"; // a quote: "
let also_doesnt_start = /* " */
"until here,
test,*/
test"; // another quote: "
}
2017-06-05 14:28:52 +00:00
2018-04-08 10:55:27 +00:00
fn foo() {
let a = 4; // /*
let b = 5;
let c = 6; // */
}
2020-05-09 17:46:13 +00:00
2022-08-14 15:22:26 +00:00
2017-06-05 14:28:52 +00:00
```
2016-01-26 20:23:21 +00:00
# Bug Reports
2020-05-09 17:46:13 +00:00
2022-08-14 15:22:26 +00:00
Please include the error message and a minimum working example
including the file or file structure.
2016-01-26 20:23:21 +00:00
2023-05-24 14:38:26 +00:00
````
2022-08-14 15:22:26 +00:00
This file crashes the program:
2016-01-26 20:23:21 +00:00
< filename >
2016-12-03 11:51:34 +00:00
```
2023-05-24 14:38:26 +00:00
< file / file structure >
```
````
2018-04-08 10:55:27 +00:00
2022-08-14 15:22:26 +00:00
[Rust's enum style]: https://github.com/rust-lang/rfcs/blob/master/text/0430-finalizing-naming-conventions.md#general-naming-conventions