Week 12: Attention and Transformer

Week 12: Attention and Transformer

Announcements

Class participation, EdSTEM and Google Folder.
(Back to Course Index Page)

Tuesday

Attention and Transformer

Attention is all you need, that paper
Textbooks
Courses slides
Courses videos

Example Paper

Attention? Attention Blog by Lilian Weng
The Illustrated Transformer Blog by Jay Alammar
ViT: Transformers for Image Recognition Paper Blog Video
ViT on EEG data paper code

Example Code

Hugging Face’s Transformers library
TensorFlow and PyTorch official tutorials
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Google Colab

Thursday

No Class Next Week

Thanksgiving break
But you can prepare for your finals if you like

Recitation, Dataset

Iris
Diabetes
Fashion-MNIST
EEGEyeNet

Recitation, Algorithm

Single algorithms, LDA, SVM, KNN, and their variations etc.
Ensemble methods, Random Forest, Boosting, Voting, and their variations etc.
Deep learning, CNN, RNN, Attention (Transformer), and their variations etc.

Classification (Supervised Learning)

Linear Discriminant Analysis (LDA)
Support vector machine (SVM)
Nearest Neighbors
Decision Trees
Ensemble Methods (Random Forest, boosting, bagging, Voting)

Regression (Supervised Learning)

Linear Regression
Ridge Regression
Lasso Regression

Clustering (Unsupervised Learning)

Overview of clustering methods
K-means
Hierarchical clustering
DBSCAN