+
j R t ^ RIHt ^ RIHtHtHt ^ RIt^RIH t H
t
]' d ] ! 4 ' d ^ RI H t R R lt
R R ltR
R lt]
R R
l4 t]
RR R ll4 tR# )z,
Needed utilities for torchao FP8 training.
)partial)
TYPE_CHECKINGCallableOptionalN)is_torchao_availabletorchao_required)Float8LinearConfigc L V ^8 d QhR\ P P /# modeltorchnnModule)formats "T/home/luca/screen-checking/.venv/lib/python3.14/site-packages/accelerate/utils/ao.py __annotate__r s % % % c RRr!V P 4 F9 w r4\ V\ P P 4 ' g K1 Vf TpTpK; W3# )z
Finds the first and last linear layer names in a model.
This is needed during FP8 to avoid issues with instability by keeping the first and last layers unquantized.
Ref: https://x.com/xariusrke/status/1826669142604141052
N)
named_modules
isinstancer r Linear)r first_linear last_linearnamemodules & r find_first_last_linear_layersr sP !%d+ ++- fehhoo . .# # K .
$$r c R V ^8 d QhR\ R\ \ , R\ /# )r fqnlayers_to_filterreturn)strlistbool)r s "r r r 1 s% c T#Y 4 r c \ V \ P P 4 ' d3 V P ^, ^ 8w g V P
^, ^ 8w d R# W9 d R# R# )an
A function which will check if `module` is:
- a `torch.nn.Linear` layer
- has in_features and out_features divisible by 16
- is not part of `layers_to_filter`
Args:
module (`torch.nn.Module`):
The module to check.
fqn (`str`):
The fully qualified name of the layer.
layers_to_filter (`List[str]`):
The list of layers to filter.
FT)r r r r in_features out_features)r r r s &&&r filter_linear_layersr( 1 sM &%((//** "a '6+>+>+Cq+H
r c 0 V ^8 d QhR\ R\ /# )r r r! )r" r$ )r s "r r r H s [ [S [T [r c 8 \ V 4 w r#\ WW#.R7 # )av
A filter function which will filter out all linear layers except the first and last.
For stability reasons, we skip the first and last linear layers Otherwise can lead to the model not training or
converging properly
Args:
module (`torch.nn.Module`):
The module to check.
fqn (`str`):
The fully qualified name of the layer.
r )r r( )r r r r s && r #filter_first_and_last_linear_layersr, H s! " !>f E L |>Y ZZr c L V ^8 d QhR\ P P /# r
r
)r s "r r r ^ s r c l ^ RI Hp V P 4 F w r#\ W14 ' g K R# R# ) ) Float8LinearTF) torchao.float8.float8_linearr0 r r )r r0 r r s & r
has_ao_layersr2 ] s- 9 ++- f + + . r c V ^8 d QhR\ P P R\ R, R\ \ , /# )r r configr module_filter_func)r r r r r )r s "r r r h s> 'Z 'Z 88??'Z ) *'Z !*'Zr c l ^ RI Hp \ V 4 w rEVf \ \ WE.R7 pV! WVR7 R# )a
Converts all `nn.Linear` layers in the model (except the first and last) to torchao's `Float8Linear` layer inplace.
Args:
model (`torch.nn.Module`):
The model to convert.
config (`torchao.float8.Float8LinearConfig`, *optional*):
The configuration for the FP8 training. Recommended to utilize
`torchao.float8.recipe_name_to_linear_config` to generate this. In general, the default config should be
sufficient (what is passed when set to `None`).
module_filter_func (`Callable`, *optional*, defaults to `filter_linear_layers`):
Optional function that must take in a module and layer name, and returns a boolean indicating whether the
module should be converted to FP8. Defaults to `filter_linear_layers`. See it for an example.
Example:
```python
from accelerate.utils.ao import convert_model_to_fp8_ao
from accelerate import Accelerator
accelerator = Accelerator(
model = MyModel()
model.to(accelerator.device)
convert_to_float8_training(model)
model.train()
```
)convert_to_float8_trainingNr+ )module_filter_fnr4 )torchao.float8r7 r r r( )r r4 r5 r7 r r s &&& r convert_model_to_fp8_aor: g s7 F : =e D L! $%9\Lg h uRXYr )NN)__doc__ functoolsr typingr r r r importsr r r1 r r r( r, r2 r: r r r@ sf 4 4 ; C%".[* 'Z 'Zr