css parsing中词法的RegEx（python）

昊漫玉

　　W3C网站上的有点小bug，而且他们写的正则是FLEX一类的，和自己手写时不太一样。瞎改了一下，贴这，谁用得着谁拿去，有问题请通知作者更新。虽然是python的，其实现在这些RegEx都是perl风格的，应该能通用。
　　除了做词法，这些定义也可以这么用：如果是找找特定字串，从这里选取一部分改吧改吧就好了；或者在这堆天书里看看有啥用法是咱不知道的？
　　(?:..)的意思是不建立capturing group，不知道别的运行时是不是这么定义的；字串前面的r意思是不用转义字符，.NET里好像是@？已然忘了。另外，W3C的词法分析配置是忽略大小写，不忽略空白：别忘了后代选择器是基于空白的。另外，w3c的没有.class的定义，这是我自己加的，会让微软的filter变成：IDENT CLASS CLASS LPARENTHESE..这样（本来w3c版本也解析不了这个），其它副作用不明。
　　（版本：CSS 2.1 + CSS 3 Selector，问我为啥不80字符换行？不乐意。）




A = r"(?:a|\\0{0,4}(?:41|61)(?:\r\n|[ \t\r\n\f])?)"
C = r"(?:c|\\0{0,4}(?:43|63)(?:\r\n|[ \t\r\n\f])?)"
D = r"(?:d|\\0{0,4}(?:44|64)(?:\r\n|[ \t\r\n\f])?)"
E = r"(?:e|\\0{0,4}(?:45|65)(?:\r\n|[ \t\r\n\f])?)"
G = r"(?:g|\\0{0,4}(?:47|67)(?:\r\n|[ \t\r\n\f])?|\\g)"
H = r"(?:h|\\0{0,4}(?:48|68)(?:\r\n|[ \t\r\n\f])?|\\h)"
I = r"(?:i|\\0{0,4}(?:49|69)(?:\r\n|[ \t\r\n\f])?|\\i)"
K = r"(?:k|\\0{0,4}(?:4b|6b)(?:\r\n|[ \t\r\n\f])?|\\k)"
L = r"(?:l|\\0{0,4}(?:4c|6c)(?:\r\n|[ \t\r\n\f])?|\\l)"
M = r"(?:m|\\0{0,4}(?:4d|6d)(?:\r\n|[ \t\r\n\f])?|\\m)"
N = r"(?:n|\\0{0,4}(?:4e|6e)(?:\r\n|[ \t\r\n\f])?|\\n)"
O = r"(?:o|\\0{0,4}(?:4f|6f)(?:\r\n|[ \t\r\n\f])?|\\o)"
P = r"(?:p|\\0{0,4}(?:50|70)(?:\r\n|[ \t\r\n\f])?|\\p)"
R = r"(?:r|\\0{0,4}(?:52|72)(?:\r\n|[ \t\r\n\f])?|\\r)"
S = r"(?:s|\\0{0,4}(?:53|73)(?:\r\n|[ \t\r\n\f])?|\\s)"
T = r"(?:t|\\0{0,4}(?:54|74)(?:\r\n|[ \t\r\n\f])?|\\t)"
U = r"(?:u|\\0{0,4}(?:55|75)(?:\r\n|[ \t\r\n\f])?|\\u)"
X = r"(?:x|\\0{0,4}(?:58|78)(?:\r\n|[ \t\r\n\f])?|\\x)"
Z = r"(?:z|\\0{0,4}(?:5a|7a)(?:\r\n|[ \t\r\n\f])?|\\z)"
# from css 3 selector
#D = r"d|\\0{0,4}(44|64)(\r\n|[ \t\r\n\f])?"
#E = r"e|\\0{0,4}(45|65)(\r\n|[ \t\r\n\f])?"
#N = r"n|\\0{0,4}(4e|6e)(\r\n|[ \t\r\n\f])?|\\n"
#O = r"o|\\0{0,4}(4f|6f)(\r\n|[ \t\r\n\f])?|\\o"
#T = r"t|\\0{0,4}(54|74)(\r\n|[ \t\r\n\f])?|\\t"
#V = r"v|\\0{0,4}(58|78)(\r\n|[ \t\r\n\f])?|\\v"

s = r"[ \t\r\n\f]+"
w = r"[ \t\r\n\f]*"
nl = r"(?:\n|\r\n|\r|\f)"
num = "\-?(?:[0-9]+|[0-9]*\.[0-9]+)"
nonascii = r"[^\0-\237]" # 2.1: [\200-\4177777]
unicode_ = r"\\[0-9a-f]{1,6}(?:\r\n|[ \n\r\t\f])?"
escape = "(?:" + unicode_ + r"|\\[^\n\r\f0-9a-f])"
string1 = r'\"(?:[^\n\r\f\\"]|\\' + nl + r'|' + escape + ')*\"'
string2 = r"\'(?:[^\n\r\f\\']|\\" + nl + r"|" + escape + r")*\'"
string = "(?:" + string1 + "|" + string2 + ")"
invalid1 =  r'\"(?:[^\n\r\f\\"]|\\' + nl +'|' + nonascii + '|' + escape + ')*'
invalid2 =  r"\'(?:[^\n\r\f\\']|\\" + nl + "|" + nonascii + "|" + escape + ")*"
invalid =   invalid1 + "|" + invalid2
comment = r"\/\*[^*]*\*+(?:[^/*][^*]*\*+)*\/"
badcomment1 = r"\/\*[^*]*\*+(?:[^/*][^*]*\*+)*"
badcomment2 = r"\/\*[^*]*(?:\*+[^/*][^*]*)*"
badcomment = r"(?:" + badcomment1 + "|" + badcomment2 + ")"
nmchar = r"(?:[_a-z0-9\-]|" + nonascii + "|" + escape + ")"
nmstart = r"(?:[_a-z]|" + nonascii + "|" + escape + ")"
ident = r"[-]?" + nmstart + "(?:" + nmchar + ")*"
name = nmchar + "+"
unicode_range = r"u\+[0-9a-f?]{1,6}(?:-[0-9a-f]{1,6})?"
openbrace = r"\{"
closingbrace = r"\}"
openparenthese = r"\("
closingparenthese = r"\)"
openbracket = r"\["
closingbracket = r"\]"
url = r"(?:[\!\#\$\%\&\*-\~]|" + nonascii + "|" + escape + ")*" # from http://www.w3.org/TR/CSS21/grammar.html

uri = r"url\(" + w + string + w + r"\)" + r"|url\(" + w + r"(?:[\!\#\$\%\&\*-\[\]-\~]|" + nonascii + r"|" + escape + r")*" + w + r"\)"
baduri1 = r"url\(" + w + "(?:[\!\#\$\%\&\*-\[\]-\~]|" + nonascii + "|" + escape + ")*" + w
baduri2 = r"url\(" + w + string + w
baduri3 = r"url\(" + w + invalid
baduri = "(?:" +baduri1 + "|"  + baduri2 + "|" + baduri3 + ")"
NOT = N+O+T
IMPORT = I+M+P+O+R+T
PAGE = P+A+G+E
MEDIA = M+E+D+I+A
IMPORTANT = I+M+P+O+R+T+A+N+T
URL = U+R+L
words = (
    ("S", s),
    ("COMMENT", comment),
    ("BADCOMMENT", badcomment),
    ("CDO", ""),
    ("INCLUDES", r"\~\="),
    ("DASHMATCH", r"\|\="),
    ("PREFIXMATCH", r"\^\="),
    ("SUFFIXMATCH", r"\$\="),
    ("SUBSTRINGMATCH", r"\*\="),   
    ("STRING", string),
    ("BADSTRING", invalid),
    ("URI", "(?:" + URL + r"\(" + w + string + w + r"\)|" + URL + r"\(" + w + url + w + r"\))"),
    ("BAD_URI", baduri),
    ("FUNCTION", ident + r"\("),
    ("IDENT", ident),
    ("HASH", r"\#" + name),
    ("CLASS", r"\." + ident),   
    ("IMPORT_SYM", r"\@" + IMPORT),
    ("PAGE_SYM", r"\@" + PAGE),
    ("MEDIA_SYM", r"\@" + MEDIA),
    ("CHARSET_SYM", r"\@charset "),
    ("IMPORTANT_SYM", r"\!(" + w + "|" + comment + ")*" + IMPORTANT),
    ("EMS", num + E + M),
    ("EXS", num + E + X),
    ("LENGTH", "(?:" + num + P + X + "|" + num + C + M + "|" + num + M + M + "|" + num + I + N + "|" + num + P + T + "|" + num + P + C + ")"),
    ("ANGLE", "(?:" + num + D + E + G + "|" + num + R + A + D + "|" + num + G + R + A + D + ")"),
    ("TIME", "(?:" + num + M + S + "|" + num + S + ")"),   
    ("FREQ", "(?:" + num + H + Z + "|" + num + K + H + Z + ")"),
    ("DIMENSION", num + ident),
    ("PERCENTAGE", num + r"\%"),
    ("NUMER", num),   
    #("FUNCTION", ident + r"\("),
    ("PLUS", w + r"\+"),
    ("GREATER", w + r"\>"),
    ("COMMA", w + r"\,"),
    ("TILDE", w + r"\~"),
    ("NOT", ":" + NOT + r"\("),
    ("ATKEYWORD", r"\@" + ident),
    ("SEMICOLON", r"\;"),
    ("COLON", r"\:"),
    ("LBRACE", openbrace),
    ("RBRACE", closingbrace),
    ("RPARENTHESE", closingparenthese),
    ("LPARENTHESE", openparenthese),
    ("LBRACKET", openbracket),
    ("RBRACKET", closingbracket),
    ("EQUALSIGN", r"\="),
    ("PERIOD", r"\."),
    ("STAR", r"\*"),   
    ("ANY", ".")
)
　　简单说明一下，如果用RegEx库做词法分析，我是把这些用“|”连起来，然后不停的match。现在RegEx都是回溯引擎，至少对于Python来说，放前面的会被优先test，所以具有相同“First集”的，越特殊越要放前面。比如num，LENGTH以它开头，NUMBER就是num，那么LENGTH的放前头。FUNCTION、URL之类同理。对于NFA/DFA引擎或等价（或严格的超集）的标准正则表达式库（!=RegEx），会leftmost longest，所以应该无需关心顺序。
　　最后那个ANY是干嘛？没法match的全都ANY。解析CSS很少需要严格要求用户的文件，不认识的不理就算了。词法上是这样，语法的时候更容易处理。
　　最后骂一句，我怎么干上程序员这行当了，弄这些繁琐的旁枝末节真TM无聊的紧。还有，早知道要处理整套页面应用相关问题，绝B用C/C++写了，从webkit里直接copy，还能顺便学点其它不相干的经验，日了。