LLM Study Diary #3: PyTorch

Continuation of the course...This lesson talks a lot related to pytorch. Tensor Basics & Memory It talks about the tensors as the core building blocks for parameters, gradients, and optimizer states. And then he discusses floating-point representations, including FP32 (full precision), BF16 (brain float, often preferred for deep learning), and the move toward FP8 for efficiency Float Data Types There are many float types have been discussed, such as float32, float 16, bfloat16, fp8, etc. Usi