Technical Screen: Explain the Transformer Architecture

Scope

Provide a structured deep-dive into Transformers. Your explanation should cover theory, shapes/equations, engineering considerations, and practical adaptations to molecular data.

Required Topics

1. Encoder/decoder stack
   • Encoder blocks and decoder blocks (where self-attention, cross-attention, and position-wise feed-forward networks fit)
   • Residual connections and normalization placement
2. Attention mechanisms
   • Self-attention vs cross-attention
   • Scaled dot-product attention: equation and tensor shapes for queries (Q), keys (K), and values (V)
   • Multi-head attention: how heads are formed and concatenated
3. Positional information
   • Absolute positional encodings: sinusoidal vs learned
   • Relative position methods (e.g., relative biases, rotary encodings) and their impact on order/generalization
4. Model families and masking
   • Encoder-only vs decoder-only vs encoder–decoder models
   • Masking strategies for autoregressive decoding (causal mask, padding mask)
5. Complexity and scaling
   • Time/memory cost O(n²) of attention and practical inference details (KV cache)
   • Methods to handle long sequences: sparse and linear-attention variants; trade-offs
6. Stability and initialization
   • LayerNorm placement (pre-LN vs post-LN), residual connections, stability considerations
   • Initialization and other training practices (dropout, LR warmup, etc.)
7. Adapting to molecular data
   • SMILES: tokenization, stereochemistry handling, data augmentation
   • Molecular graphs: inputs/features, positional/edge encodings
   • Training objectives: masked LM, autoregressive LM, contrastive pretraining