Nested parens are required to allow permutations of words in a trigger. Regular expressions handle them just fine, however the parser.check_syntax() method in the Python version (not in JS) is checking for nested parens (and other things) and generating a bogus "Unbalanced brackets" message (which is misleading). I recommend either removing this check entirely, or just removing parens from the list (and from the total).
Sample valid trigger: ((cat|dog) [*] animal|animal [*] (cat|dog))