Add (already referenced) initial docs for tokens

Change-Id: I8edb453f638223464ffb8563c8d4fe0dd6999991
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/269360
Reviewed-by: Samuel Rawlins <srawlins@google.com>
Commit-Queue: Brian Wilkerson <brianwilkerson@google.com>

pkg/analyzer/doc/tutorial/tokens.md

@@ -0,0 +1,13 @@
+# The Token stream
+
+The token stream is a doubly linked list with special 'EOF' tokens at the
+beginning and end. The EOF token at the end points to itself as the `next`
+token, and the one at the beginning points to itself as the `previous` token.
+So, for a correctly formed token stream (which we routinely assume will always
+be the case) neither `previous` nor `next` can ever return `null`. We can't
+express that in the type system because those fields will be `null` temporarily
+while the list is being built. As a result, the convention throughout the code
+base, is to suffix those two getters with a bang (`!`) everywhere they're
+invoked.
+
+Tokens are typically accessed through the [AST](ast.md).