• Serhiy Storchaka's avatar
    bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370) · 8ac65811
    Serhiy Storchaka authored
    "Include/token.h", "Lib/token.py" (containing now some data moved from
    "Lib/tokenize.py") and new files "Parser/token.c" (containing the code
    moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
    in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
    "Tools/scripts/generate_token.py". The script overwrites files only if
    needed and can be used on the read-only sources tree.
    
    "Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
    instead of been executable itself.
    
    Added new make targets "regen-token" and "regen-symbol" which are now
    dependencies of "regen-all".
    
    The documentation contains now strings for operators and punctuation tokens.
    8ac65811
tokenizer.c 49.4 KB