Llama 2
https://arxiv.org/abs/2307.09288Open Foundation and Fine-Tuned Chat Models
https://ai.meta.com/llama/#partnershipsIntroducing Llama 2
https://github.com/facebookresearch/llama/tree/mainfacebookresearch/llama
https://github.com/facebookresearch/llama-recipes/Llama 2 Fine-tuning / Inference Recipes and Examples
https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/Llama 2: Open Foundation and Fine-Tuned Chat Models
>@PhysConsultant: LLaMaは基本的なtransformerに↓の3つの工夫施したもの。
>・pre-normalization using RMSNorm
>・SwiGLU activation function
>それで、LLaMa2はさらに下記の変更を加えた。
>・Context Lengthを2倍。
>・↑でメモリ空間が巨大になるためGrouped-Query Attentionを採用。
ライセンス