mathdatech/Python
1
0
Fork 0
mirror of https://github.com/TheAlgorithms/Python.git synced 2025-02-17 14:58:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

Added A General Swish Activation Function inNeural Networks (#10415)

Browse Source 
* Added A General Swish Activation Function inNeural Networks

* Added the general swish function in the SiLU function and renamed it as swish.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Shivansh Bhatnagar <shivansh.bhatnagar.mat22@iitbhu.ac.in>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Shivansh Bhatnagar 2023-10-18 20:20:18 +05:30 committed by GitHub
parent 361f64c21d
commit 572de4f15e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 20 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
neural_network/activation_functions/sigmoid_linear_unit.py → neural_network/activation_functions/swish.py
View File

@ -12,6 +12,7 @@ image classification and machine translation.
This script is inspired by a corresponding research paper. This script is inspired by a corresponding research paper.
* https://arxiv.org/abs/1710.05941 * https://arxiv.org/abs/1710.05941
* https://blog.paperspace.com/swish-activation-function/
""" """
import numpy as np import numpy as np
@ -49,6 +50,25 @@ def sigmoid_linear_unit(vector: np.ndarray) -> np.ndarray:
return vector * sigmoid(vector) return vector * sigmoid(vector)
def swish(vector: np.ndarray, trainable_parameter: int) -> np.ndarray:
"""
Parameters:
vector (np.ndarray): A numpy array consisting of real values
trainable_parameter: Use to implement various Swish Activation Functions
Returns:
swish_vec (np.ndarray): The input numpy array, after applying swish
Examples:
>>> swish(np.array([-1.0, 1.0, 2.0]), 2)
array([-0.11920292, 0.88079708, 1.96402758])
>>> swish(np.array([-2]), 1)
array([-0.23840584])
"""
return vector * sigmoid(trainable_parameter * vector)
if __name__ == "__main__": if __name__ == "__main__":
import doctest import doctest