mathdatech/Python
1
0
Fork 0
mirror of https://github.com/TheAlgorithms/Python.git synced 2025-04-23 22:23:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add files via upload

Browse Source
This commit is contained in:
Pritam Das 2024-10-20 22:05:16 +05:30 committed by GitHub
parent b23cc1a515
commit b3c2a73a10
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
neural_network/sliding_window_attention.py
View File

@ -1,6 +1,6 @@
""" """
- - - - - -- - - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - - - - - - - - - - - - - - - - - - - -
Name - - Sliding Window Attention Mechanism Name - - sliding_window_attention.py
Goal - - Implement a neural network architecture using sliding window attention for sequence modeling tasks. Goal - - Implement a neural network architecture using sliding window attention for sequence modeling tasks.
Detail: Total 5 layers neural network Detail: Total 5 layers neural network
* Input layer * Input layer
@ -13,6 +13,7 @@ Date: 2024.10.20
References: References:
1. Choromanska, A., et al. (2020). "On the Importance of Initialization and Momentum in Deep Learning." *Proceedings of the 37th International Conference on Machine Learning*. 1. Choromanska, A., et al. (2020). "On the Importance of Initialization and Momentum in Deep Learning." *Proceedings of the 37th International Conference on Machine Learning*.
2. Dai, Z., et al. (2020). "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention." *arXiv preprint arXiv:2006.16236*. 2. Dai, Z., et al. (2020). "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention." *arXiv preprint arXiv:2006.16236*.
3. [Attention Mechanisms in Neural Networks](https://en.wikipedia.org/wiki/Attention_(machine_learning))
- - - - - -- - - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - - - - - - - - - - - - - - - - - - - -
""" """