Skip to content
This repository was archived by the owner on Aug 20, 2025. It is now read-only.

Conversation

@ottobackwards
Copy link
Contributor

@ottobackwards ottobackwards commented Oct 24, 2017

as discussed on the dev list: Stellar support for switch/case style conditionals

This PR adds 'match' capability to the stellar language. This is very similar to the Scala match feature.

FROM THE README:

Stellar Language Match Expression

Stellar provides the capability to write match expressions, which are similar to switch statements commonly found in c like languages, but more like
Scala's match.

The syntax is:

  • match{ logical_expression1 => evaluation expression1, logical_expression2 => evaluation_expression2, default => default_expression}

Where:

  • logical_expression is a Stellar expression that evaluates to true or false. For instance var > 0 or var > 0 AND var2 == 'foo' or IF ... THEN ... ELSE
  • evaluation_expression is any Stellar Expression
  • default is a required default return value, should no logical expression match

default is required

Lambda expressions are supported, but they must be no argument lambdas such as () -> STATEMENT

  • Only the first clause that evaluates to true will be executed.

Review items

  • more test cases
  • help with issue with MAP() function
  • correctness of short circuit / grammar

This PR does not

  • add the aliasing of long variables
  • support variable arg lambda

Testing

from the Stellar shell execute various statements involving match such as:

ttofowler@Winterfell [13:01:16] [~/src/apache/forks/metron/metron-stellar/stellar-common] [stellar_match]
-> % mvn exec:java \
   -Dexec.mainClass="org.apache.metron.stellar.common.shell.StellarShell" -Dexec.args="-l src/test/resources/log4j.properties"
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building stellar-common 0.4.1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ stellar-common ---
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> foo := 1
[Stellar]>>> match{foo == 1 : TO_UPPER('ok')}
OK
[Stellar]>>>

For all changes:

  • Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
  • Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
  • Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

  • Have you included steps to reproduce the behavior or problem that is being changed or addressed?

  • Have you included steps or a guide to how the change may be verified and tested manually?

  • Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:

    mvn -q clean integration-test install && build_utils/verify_licenses.sh 
    
  • Have you written or updated unit tests and or integration tests to verify your changes?

  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

  • Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:

    cd site-book
    mvn site
    

Scala's match.

The syntax is:
* `match{ logical_expression1 : evaluation expression1, logical_expression2 : evaluation_expression2` : A match expression with no default
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a missing } here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, thanks

NULL : 'null' | 'NULL';
NAN : 'NaN';

AS : 'as' | 'AS';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this may be fairly unlikely, but technically this is a breaking change for Stellar. I believe if anyone had a variable called as, AS, etc. it would fail with a parse exception. I think we should explicitly call this out. Also, is this something we technically have to wait for a major version release to add?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, AS was for the match(longVarName as shortName){ x > 5 : shortName is available }
case, but since I'm not doing that, i should remove it.

@Ignore
public void testMatchMAPEvaluation() {

// NOTE: THIS IS BROKEN RIGHT NOW.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you delete this comment and add it to @Ignore("NOTE: THIS IS BROKEN RIGHT NOW.") instead? Maybe also mention that it's broken because the use of MAP is not supported right now. It may even be better to just delete the test, as it will probably be written when that feature is implemented. Also, this may just be forgotten about.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually wanted to leave this in so @cestella might help me figure out why it isn't working with map. How about I leave it for now, and you hold your plus +1 until it is removed. I would got to commit with this, but I want it there for the review.


@Test
@SuppressWarnings("unchecked")
public void testShortCircut() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor spelling error (Circut -> Circuit).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@SuppressWarnings("unchecked")
public void testMatchErrorNoDefault() {

boolean caught = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't use a boolean flag to determine if an exception was thrown. If you just want to expect an exception we should use @Test(expected = ParseException.class). For more indepth testing we should use @Rule public ExpectedException thrown = ExpectedException.none();.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. When I was developing, I just had one big test function that I was adding test after test too, so I used this to catch the exception and keep going. When I did the PR I refactored the tests, but forgot to clean this up. Thanks!


Where:

* `logical_expression` is a Stellar expression that evaluates to true or false. For instance `var > 0` or `var > 0 AND var2 == 'foo'`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems as if the below stellar always throws an exception. Is it by design that default clauses are always evaluated? If so can you mention that here?

foo := 500
match{ foo < 100 : THROW('oops'), foo > 200 : 'ok', default : THROW('exception thrown') }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...

[Stellar]>>> foo := 500
[Stellar]>>> match{ foo < 100 : THROW('oops'), foo > 200 : 'ok', default : THROW('exception thrown') }
ok
[Stellar]>>>

I don't see that.

Also, if you check the last test in TestMatch you will see I have a test for this kind of thing.

}
if (skipMatchClauses && (token.getUnderlyingType() == MatchClauseEnd.class
|| token.getUnderlyingType() == MatchClauseCheckExpr.class)) {
while (it.hasNext()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will only skip one match clause after a true one is found. See below test for a stellar statement that fails with a ParseException.

@Test
@SuppressWarnings("unchecked")
public void moreThanTwoConsecutiveTrueCasesWillResultInFailure() {
    Assert.assertEquals("ok",  run("match{ foo < 100 : THROW('oops'), foo > 200 : 'ok', foo > 300 : 'ok', default : 'works' }",
            new HashMap() {{
                put("foo", 500);
            }}));
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is another issue with validate

transformation_expr
;

match_clause_check :
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this extra expression needed? Could we get rid of it? Instead of this just have what's below?

match_clause_action :
  transformation_expr #MatchClauseAction
  ;

match_clause_check :
  logical_expr #MatchClauseCheckExpr
  ;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try that again. I don't remember the specific reason why I ended up with it factored like this. But this is what ended up working ;)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

### Stellar Language Match Expression

Stellar provides the capability to write match expressions, which are similar to switch statements commonly found in c like languages, but more like
Scala's match.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be a bit hesitant to say this is similar to Scala's match. Our match statement doesn't really support pattern matching.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any suggestions for a better way to describe it? I'm not super happy with this either, I'll take suggestions ;)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ottobackwards sure. Right now, I think our match statement syntax is just a nicer syntax for if then else blocks. I would just state something along those lines. I know in the future it will be more.

@jjmeyer0
Copy link
Contributor

Also, I don't think there is an issue with the map function directly. It's actually an issue with the logical_expression var1 and using/not using a default branch. It seems like the stack isn't properly being managed. For example things are being pushed (eg. when a logical_expr is just var1) when they shouldn't and popped when they shouldn't be (eg. when there is no default?). I haven't pin pointed it directly yet, but I hope this helps somewhat. Below are a few example tests that I have created.

// Here there is a null on the stack
@Test
@SuppressWarnings("unchecked")
public void test1FailsWithMoreOnStack() {
    Assert.assertEquals("a",  run("match{ foo : 'a' }",
            new HashMap() {{
                put("foo", true);
            }}));
}

// Here it's an empty stack when we expect 'a' i believe.

@Test
@SuppressWarnings("unchecked")
public void test1FailsEmptyStack() {
    Assert.assertEquals("a",  run("match{ foo == true : 'a' }",
            new HashMap() {{
                put("foo", true);
            }}));
}
// This is *not* properly handled
@Test
@SuppressWarnings("unchecked")
public void test1Passes() {
    Assert.assertEquals("a",  run("match{ foo : 'a', default: 'b' }",
            new HashMap() {{
                put("foo", true);
            }}));
}
// This is properly handled
@Test
@SuppressWarnings("unchecked")
public void test1Passes() {
    Assert.assertEquals("a",  run("match{ foo == true : 'a', default: 'b' }",
            new HashMap() {{
                put("foo", true);
            }}));
}
// Example where map works
@Test
@SuppressWarnings("unchecked")
public void workingMatchWithMap() {
   Assert.assertEquals(Arrays.asList("OK", "HAHA"),  run("match{ foo > 100 : THROW('oops'), foo > 200 : THROW('oh no'), foo >= 50 : MAP(['ok', 'haha'], (a) -> TO_UPPER(a)), default: 'a' }",
          new HashMap() {{
            put("foo", 50);
          }}));
}

@ottobackwards
Copy link
Contributor Author

Thanks, I added those tests to the test class, and these fail:

@Test
  @SuppressWarnings("unchecked")
  public void testVariableOnlyNoDefault() {
    Assert.assertEquals("a",  run("match{ foo : 'a' }",
        new HashMap() {{
          put("foo", true);
        }}));
  }

  @Test
  @SuppressWarnings("unchecked")
  public void testVariableEqualsCheckNoDefault() {
    Assert.assertEquals("a",  run("match{ foo == true : 'a' }",
        new HashMap() {{
          put("foo", true);
        }}));
  }

  @Test
  @SuppressWarnings("unchecked")
  public void testVariableOnlyCheckWithDefault() {
    Assert.assertEquals("a",  run("match{ foo : 'a', default: 'b' }",
        new HashMap() {{
          put("foo", true);
        }}));
  }

I will take a look. I feel that I have similar tests to these that are working, so it is strange.

@ottobackwards
Copy link
Contributor Author

@jjmeyer0 thanks for the tests. The issue is that we call validate and then parse. When we call validate, we resolve all vars to NULL. This means that the clauses in these failures are false. As stated in the readme, something in the match must be true or you get an error. In this case, we have an issue where validate is forcing us into that condition and we don't have a default, so it is an error.
I'll keep looking.
If you have any thoughts, or maybe @cestella does, I'm all ears.

@ottobackwards
Copy link
Contributor Author

So for example:

 /*
          curr is the current value on the stack.  This is the non-deferred actual evaluation for this expression
          and with the current context.
           */
          Token<?> curr = instanceDeque.peek();
          if (curr != null && curr.getValue() != null && curr.getValue() instanceof Boolean
              && ShortCircuitOp.class.isAssignableFrom(token.getUnderlyingType())) {
            //if we have a boolean as the current value and the next non-contextual token is a short circuit op
            //then we need to short circuit possibly
            if (token.getUnderlyingType() == BooleanArg.class) {

with this function:

 @Test
  @SuppressWarnings("unchecked")
  public void testVariableOnlyCheckWithDefault() {
    Assert.assertEquals("a",  run("match{ foo : 'a', default : 'b' }",
        new HashMap() {{
          put("foo", true);
        }}));
  }

When we get to that point, because of the validation of vars to null curr IS null. So even though token IS a short circuit token, we do not process it as we should.

@ottobackwards
Copy link
Contributor Author

Because it is possible in validate() for a boolean to be null instead of true or false.

…ests that throw exceptions.

Also partially addressed issues around validation during tests resulting in empty stacks or other stack issues
because of null variable resolution.  The fix is not complete.  I have added tests to cover the discovered failing
cases, and they are all resolved but one.  This one is ignored for the moment while there is review
discussion around validation
@ottobackwards
Copy link
Contributor Author

ottobackwards commented Oct 29, 2017

@jjmeyer0 @cestella : Ok.
So, I added some code, where it is possible context wise to guard against the case where there is a single variable as the check, and no default. This resolves the test issues, save for one where there is no default and we are using == true. This is more difficult.

I feel as if all of these things are wrong, because the validation by executing with null variables is itself logically wrong. We have discussed this before, but this is really evident here. At least I feel it is.

I think that we should use compilation instead of validation. We can decide to make it a separate option when running tests, or to make it the way we do all tests etc. But if compilation does what I think it does, then I think it is more correct.

I would like to do that, and remove the work around I have introduced here ( basically detecting that we are validating and have a single var that is null because of validation ) since changing validation would be the real complete answer.

@ottobackwards
Copy link
Contributor Author

for example:

 @Test
  public void testValidation() {
    Object value = run("IF x == null THEN THROW('it cannot be null') ELSE 'it is ok'", new HashMap(){{
      put("x","something");
    }} );
  }

This is a valid test, but it fails on validation not parse.

1. Require default statement
2. change the action separator from : to => so that...
3. we can use conditional_exp and logical_exp ( can check for EXISTS() with IF etc )
@ottobackwards
Copy link
Contributor Author

OK, the validation issues are a big big deal. I have changed the match functionality so that they work in that context, supporting guard expressions and requiring the default operation.

  1. Require default statement
  2. change the action separator from : to => so that...
  3. we can use conditional_exp and logical_exp ( can check for EXISTS() with IF etc )

I still have to do the hack for the match{ var1 => 'ok' , default => 'notOk' }
since the lone null as logical doesn't work as stated earlier.

So the new syntax is

match { logical_or_conditional_expr => transform_exp , default => transform_exp }

@ottobackwards
Copy link
Contributor Author

the issue with MAP() is resolved as well

@jjmeyer0
Copy link
Contributor

jjmeyer0 commented Nov 5, 2017

@ottobackwards something is still going on with this. I'm seeing the following behavior:

[Stellar]>>> foo := 500
[Stellar]>>> match{ foo > 100 => ['oops'], foo > 200 => ['oh no'], foo >= 500 => MAP(['ok', 'haha'], (a) -> TO_UPPER(a)), default => ['a']}
[!] Invalid parse, found [OK, HAHA]
org.apache.metron.stellar.dsl.ParseException: Invalid parse, found [OK, HAHA]
        at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:210)
        at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152)
        at org.apache.metron.stellar.common.shell.StellarExecutor.execute(StellarExecutor.java:292)
        at org.apache.metron.stellar.common.shell.StellarShell.handleStellar(StellarShell.java:282)
        at org.apache.metron.stellar.common.shell.StellarShell.execute(StellarShell.java:514)
        at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[Stellar]>>> 

@ottobackwards
Copy link
Contributor Author

@jjmeyer0 I have fixed the issue. I had lost short circuit to the end of all clauses, fixing that fixes the case. I added tests for your issues.

I think I'm going to refactor the tests, but you should be able to check where it is in the mean time.
Thanks for the report!

@ottobackwards
Copy link
Contributor Author

That fix also fixes the issue where IF THEN ELSE in the action clause was not working

@ottobackwards
Copy link
Contributor Author

Bump?

Copy link
Contributor

@jjmeyer0 jjmeyer0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about the delay. Things seems to be working now. I made one last comment.

| EXISTS LPAREN IDENTIFIER RPAREN #ExistsFunc
| LPAREN conditional_expr RPAREN #condExpr_paren
| functions #func
| DEFAULT #Default
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed this. Why do we have default here? This would allow default to be used in some unintended places. Is there some reasoning behind this?

@ottobackwards
Copy link
Contributor Author

Great catch! Moved it out.

@jjmeyer0
Copy link
Contributor

@ottobackwards this looks good to me now. I re-built everything, ran the tests, and played around in stellar shell. The only question I'm still not sure about is the one that deals with backwards compatibility. I believe with the new keywords (default, match) we can potentially break expressions with this change. @ottobackwards and @cestella do you have an opinion on this?

@ottobackwards
Copy link
Contributor Author

Great! I am not sure on that. @cestella what do you think?

@ottobackwards
Copy link
Contributor Author

Maybe we need a tool that tests stellar scripts or configurations that the user can run and report?
And we can maintain that per release?

@cestella
Copy link
Member

@jjmeyer0 @ottobackwards Yeah, it's definitely possible to break expressions that include variables named match and default, but that's the danger that I think we have to take for expanding the language. We should make sure we update the Upgrading.md document with a brief blurb when this PR gets in.

We probably DO want a script as @ottobackwards suggests, but I don't think it's required here for this PR. All of this is my $0.02

@ottobackwards
Copy link
Contributor Author

Ok, I'll update the upgrading doc.

@ottobackwards
Copy link
Contributor Author

ottobackwards commented Nov 29, 2017

Ok, please have a look at the doc. I'll note that we have not added any notes previously for this type of thing.

Also: METRON-1339: Stellar shellShould have a way to validate deployed functions has been created.
You are welcome to comment.

I may have a PR for that soon

@jjmeyer0
Copy link
Contributor

once travis build completes successfully this looks good +1.

@ottobackwards
Copy link
Contributor Author

@cestella any last words before I pull the trigger on this? Any comment?

@cestella
Copy link
Member

Yeah, let me run this up once more in the REPL and take a final look at the docs, but I've been monitoring and I like what I see so far.

@ottobackwards
Copy link
Contributor Author

PR: #856 adds capability to use the stellar shell to validate stellar statements at rest out of ZK

@cestella
Copy link
Member

Alright, +1 otto, you did great work here. I'm very, very impressed. Thanks to @jjmeyer0 for the careful review and assistance. Open source dev at its finest.

@cestella
Copy link
Member

cestella commented Nov 30, 2017

actually, hold on there, I found one more bug:

[Stellar]>>> match { is_alert == null => null, is_alert => 'alert', default => 'nah' }
[!] null
java.lang.NullPointerException
	at org.apache.metron.stellar.common.StellarCompiler.lambda$exitMatchClauseAction$19(StellarCompiler.java:752)
	at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:190)
	at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152)
	at org.apache.metron.stellar.common.shell.StellarExecutor.execute(StellarExecutor.java:292)
	at org.apache.metron.stellar.common.shell.StellarShell.handleStellar(StellarShell.java:282)
	at org.apache.metron.stellar.common.shell.StellarShell.execute(StellarShell.java:514)
	at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I'm +1 after that's fixed.

@ottobackwards
Copy link
Contributor Author

Ok, what was is_alert in this test?

@cestella
Copy link
Member

sorry, it's not set, so it's null.

@ottobackwards
Copy link
Contributor Author

Ok, I'll check it out after i do my full dev test on validate

@ottobackwards
Copy link
Contributor Author

@cestella new test and fix is in

@cestella
Copy link
Member

Ok, my +1 stands

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants