LibWeb: Track the byte offset of an HTMLToken's position

We currently track the [line, column] position of every HTMLToken, as
this is what is needed for LibGUI's syntax highlighting. Some non-LibGUI
purposes (e.g. highlighting HTML with HTML) require a byte offset. Track
both during tokenization.
This commit is contained in:
Timothy Flynn 2023-08-25 08:11:26 -04:00 committed by Tim Flynn
parent ba4db899d4
commit fea440055a
2 changed files with 2 additions and 0 deletions

View file

@ -37,6 +37,7 @@ public:
struct Position {
size_t line { 0 };
size_t column { 0 };
size_t byte_offset { 0 };
};
struct Attribute {

View file

@ -223,6 +223,7 @@ void HTMLTokenizer::skip(size_t count)
} else {
m_source_positions.last().column++;
}
m_source_positions.last().byte_offset += m_utf8_iterator.underlying_code_point_length_in_bytes();
}
++m_utf8_iterator;
}