Skip to content

Add more Activation Function options. #75

Open
Adil-Iqbal wants to merge 6 commits intoCodingTrain:masterfrom
Adil-Iqbal:master
Open

Add more Activation Function options. #75
Adil-Iqbal wants to merge 6 commits intoCodingTrain:masterfrom
Adil-Iqbal:master

Conversation

@Adil-Iqbal
Copy link

@Adil-Iqbal Adil-Iqbal commented Feb 14, 2018

As per this discussion: #70

This pull request adds the following features to the library.

  • The duplicate static method - which duplicates matrix passed into it.
  • Unit tests for the duplicate static method.
  • Many new activation functions, including relu and leaky_relu.

My javascript auto-formatter cleaned up the code, but it obscured the relavent changes in the most recent commit. It might be more informative to look at an older commit...

lib/nn.js Outdated

// Range (-1, 1)
let softsign = new ActivationFunction(
x => x / 1 + Math.abs(x),

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

x => x / 1 + Math.abs(x), should be x => x / (1 + Math.abs(x)), right?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gosh darn it man. You are a genius.

@xxMrPHDxx
Copy link

Lol. Just exploring your code to add more activation function into my code and found that out. Now all of them should work fine

@Adil-Iqbal
Copy link
Author

Ran some more tests on XOR.
Everything works great! There was a 100% pass rate and most of them were quick.
Observations below:

  1. leaky_relu had the best balance of consistency and speed.
  2. relu was the fastest, but not consistent. Still exhibiting "dying relu" problem.
  3. relu was generating an interesting step patterns in the solution space. Here's a picture.
  4. arctan was solving the problem consistently, albeit not as fast as the others. The solution space looked like a large open field with folded corners. Here's a picture.
  5. softplus had the most difficult time. But it consistently solved the problem. Lots of fluctuations and BOLD decision making. The solution space looks like a glowing blob! Here's a picture.

I encourage everyone to check it out. It's really interesting!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants