Commit 8323f68f authored by Raymond Hettinger's avatar Raymond Hettinger

Issue 21977: Minor improvements to the regexes in the tokenizer example.

parent 5283c4e1
...@@ -1340,9 +1340,9 @@ successive matches:: ...@@ -1340,9 +1340,9 @@ successive matches::
('ASSIGN', r':='), # Assignment operator ('ASSIGN', r':='), # Assignment operator
('END', r';'), # Statement terminator ('END', r';'), # Statement terminator
('ID', r'[A-Za-z]+'), # Identifiers ('ID', r'[A-Za-z]+'), # Identifiers
('OP', r'[+*\/\-]'), # Arithmetic operators ('OP', r'[+\-*/]'), # Arithmetic operators
('NEWLINE', r'\n'), # Line endings ('NEWLINE', r'\n'), # Line endings
('SKIP', r'[ \t]'), # Skip over spaces and tabs ('SKIP', r'[ \t]+'), # Skip over spaces and tabs
] ]
tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification) tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification)
get_token = re.compile(tok_regex).match get_token = re.compile(tok_regex).match
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment