Document the parser's design.

R=johnniwinther@google.com, scheglov@google.com

Review-Url: https://codereview.chromium.org/2799043004 .

pkg/front_end/lib/src/fasta/parser/parser.dart

@@ -182,6 +182,90 @@ class FormalParameterType {
 ///
 /// Historically, we over-used identical, and when identical is used on other
 /// objects than strings, it can often be replaced by `==`.
+///
+/// ## Flexibility, Extensibility, and Specification
+///
+/// The parser is designed to be flexible and extensible. Its methods are
+/// designed to be overridden in subclasses, so it can be extended to handle
+/// unspecified language extension or experiments while everything in this file
+/// attempts to follow the specification (unless when it interferes with error
+/// recovery).
+///
+/// We achieve flexibily, extensible, and specification compliance by following
+/// a few rules-of-thumb:
+///
+/// 1. All methods in the parser should be public.
+///
+/// 2. The methods follow the specified grammar, and do not implement custom
+/// extensions, for example, `native`.
+///
+/// 3. The parser doesn't rewrite the token stream (when dealing with `>>`).
+///
+/// ### Implementing Extensions
+///
+/// For various reasons, some Dart language implementations have used
+/// custom/unspecified extensions to the Dart grammar. Examples of this
+/// includes diet parsing, patch files, `native` keyword, and generic
+/// comments. This class isn't supposed to implement any of these
+/// features. Instead it provides hooks for those extensions to be implemented
+/// in subclasses or listeners. Let's examine how diet parsing and `native`
+/// keyword is currently supported by Fasta.
+///
+/// #### Implementation of `native` Keyword
+///
+/// Both dart2js and the Dart VM have used the `native` keyword to mark methods
+/// that couldn't be implemented in the Dart language and needed to be
+/// implemented in JavaScript or C++, respectively. An example of the syntax
+/// extension used by the Dart VM is:
+///
+///     nativeFunction() native "NativeFunction";
+///
+/// When attempting to parse this function, the parser eventually calls
+/// [parseFunctionBody]. This method will report an unrecoverable error to the
+/// listener with the code [codeExpectedFunctionBody]. The listener can then
+/// look at the error code and the token and use the methods in
+/// [dart_vm_native.dart](dart_vm_native.dart) to parse the native syntax.
+///
+/// #### Implementation of Diet Parsing
+///
+/// We call it _diet_ _parsing_ when the parser skips parts of a file. Both
+/// dart2js and the Dart VM have been relying on this from early on as it allows
+/// them to more quickly compile small programs that use small parts of big
+/// libraries. It's also become an integrated part of how Fasta builds up
+/// outlines before starting to parse method bodies.
+///
+/// When looking through this parser, you'll find a number of unused methods
+/// starting with `skip`. These methods are only used by subclasses, such as
+/// [ClassMemberParser](class_member_parser.dart) and
+/// [TopLevelParser](top_level_parser.dart). These methods violate the
+/// principle above about following the specified grammar, and originally lived
+/// in subclasses. However, we realized that these methods were so widely used
+/// and hard to maintain in subclasses, that it made sense to move them here.
+///
+/// ### Specification and Error Recovery
+///
+/// To improve error recovery, the parser will inform the listener of
+/// recoverable errors and continue to parse.  An example of a recoverable
+/// error is:
+///
+///     Error: Asynchronous for-loop can only be used in 'async' or 'async*'...
+///     main() { await for (var x in []) {} }
+///              ^^^^^
+///
+/// For unrecoverable errors, the parser will ask the listener for help to
+/// recover from the error. We haven't made much progress on these kinds of
+/// errors, so in most cases, the parser aborts by skipping to the end of file.
+///
+/// Historically, this parser has been rather lax in what it allows, and
+/// deferred the enforcement of some syntactical rules to subsequent phases. It
+/// doesn't matter how we got there, only that we've identified that it's
+/// easier if the parser reports as many errors it can, but informs the
+/// listener if the error is recoverable or not.
+///
+/// Currently, the parser is particularly lax when it comes to the order of
+/// modifiers such as `abstract`, `final`, `static`, etc. Historically, dart2js
+/// would handle such errors in later phases. We hope that these cases will go
+/// away as Fasta matures.
 class Parser {
   final Listener listener;