+
"ݳj0\ ^ RI t ^ RIt^ RIHtHtHtHt ^ RIHtH t H
t
H t H t ^RI
Ht ^RIHt ]! 4 t] ! R R4 4 tR# ) N)asdict dataclassfieldfields)AnyLiteralOptionalSetUnion) resolve_path) setup_loggerc a ] tR t^t o Rt]! RRR/R7 t]! ^RR/R7 t]! RRR/R7 t]! RRR /R7 t ]! ^RR
/R7 t
]! R RR /R7 t ]! R
RR/R7 t ]! RRR/R7 t
]! R
RR/R7 t]! RRR/R7 t]! RRR/R7 t]! RRR/R7 t]! RRR/R7 t]! ]RR/R7 t]! ]RR/R7 t]! RRR/R7 t]! RRR /R7 t]! ]RR /R7 t]! RRR /R7 t]! R
RR /R7 t]! RRR /R7 t]! R
RR!/R7 t ]! R"RR#/R7 t ]! R$RR%/R7 t ]! RRR&/R7 t R' t V 3R( lR) lt!R* t"]#V 3R+ lR, l4 t$V 3R- lt%R.t&V t'R# )/
LoraConfiga}
This is the configuration class to store the configuration of a [`LoraModel`].
Args:
r (`int`):
Lora attention dimension (the "rank").
target_modules (`Optional[Union[List[str], str]]`):
The names of the modules to apply the adapter to. If this is specified, only the modules with the specified
names will be replaced. When passing a string, a regex match will be performed. When passing a list of
strings, either an exact match will be performed or it is checked if the name of the module ends with any
of the passed strings. If this is specified as 'all-linear', then all linear/Conv1D modules are chosen,
excluding the output layer. If this is not specified, modules will be chosen according to the model
architecture. If the architecture is not known, an error will be raised -- in this case, you should specify
the target modules manually.
exclude_modules (`Optional[Union[List[str], str]]`):
The names of the modules to not apply the adapter. When passing a string, a regex match will be performed.
When passing a list of strings, either an exact match will be performed or it is checked if the name of the
module ends with any of the passed strings.
lora_alpha (`int`):
The alpha parameter for Lora scaling.
lora_dropout (`float`):
The dropout probability for Lora layers.
fan_in_fan_out (`bool`):
Set this to True if the layer to replace stores weight like (fan_in, fan_out). For example, gpt-2 uses
`Conv1D` which stores weights like (fan_in, fan_out) and hence this should be set to `True`.
bias (`str`):
Bias type for LoRA. Can be 'none', 'all' or 'lora_only'. If 'all' or 'lora_only', the corresponding biases
will be updated during training. Be aware that this means that, even when disabling the adapters, the model
will not produce the same output as the base model would have without adaptation.
use_rslora (`bool`):
When set to True, uses Rank-Stabilized LoRA which
sets the adapter scaling factor to `lora_alpha/math.sqrt(r)`, since it was proven to work better.
Otherwise, it will use the original default value of `lora_alpha/r`.
modules_to_save (`List[str]`):
List of modules apart from adapter layers to be set as trainable and saved in the final checkpoint.
init_lora_weights (`bool` | `Literal["gaussian", "eva", "olora", "pissa", "pissa_niter_[number of iters]", "loftq"]`):
How to initialize the weights of the adapter layers. Passing True (default) results in the default
initialization from the reference implementation from Microsoft. Passing 'gaussian' results in Gaussian
initialization scaled by the LoRA rank for linear and layers. Setting the initialization to False leads to
completely random initialization and is discouraged. Pass `'loftq'` to use LoftQ initialization. Passing
`'eva'` results in a data-driven initialization of Explained
Variance Adaptation. EVA initalizes LoRA based on the SVD of layer input activations and achieves SOTA
performance due to its ability to adapt to the finetuning data. Pass `'olora'` to use OLoRA initialization.
Passing `'pissa'` results in the initialization of Principal
Singular values and Singular vectors Adaptation (PiSSA), which converges more rapidly than LoRA and
ultimately achieves superior performance. Moreover, PiSSA reduces the quantization error compared to QLoRA,
leading to further enhancements. Passing `'pissa_niter_[number of iters]'` initiates Fast-SVD-based PiSSA
initialization, where `[number of iters]` indicates the number of subspace iterations to perform FSVD, and
must be a nonnegative integer. When `[number of iters]` is set to 16, it can complete the initialization of
a 7B model within seconds, and the training effect is approximately equivalent to using SVD.
layers_to_transform (`Union[List[int], int]`):
The layer indices to transform. If a list of ints is passed, it will apply the adapter to the layer indices
that are specified in this list. If a single integer is passed, it will apply the transformations on the
layer at this index.
layers_pattern (`Optional[Union[List[str], str]]`):
The layer pattern name, used only if `layers_to_transform` is different from `None`. This should target the
`nn.ModuleList` of the model, which is often called `'layers'` or `'h'`.
rank_pattern (`dict`):
The mapping from layer names or regexp expression to ranks which are different from the default rank
specified by `r`.
alpha_pattern (`dict`):
The mapping from layer names or regexp expression to alphas which are different from the default alpha
specified by `lora_alpha`.
megatron_config (`Optional[dict]`):
The TransformerConfig arguments for Megatron. It is used to create LoRA's parallel linear layer. You can
get it like this, `core_transformer_config_from_args(get_args())`, these two functions being from Megatron.
The arguments will be used to initialize the TransformerConfig of Megatron. You need to specify this
parameter when you want to apply LoRA to the ColumnParallelLinear and RowParallelLinear layers of megatron.
megatron_core (`Optional[str]`):
The core module from Megatron to use, defaults to `"megatron.core"`.
loftq_config (`Optional[LoftQConfig]`):
The configuration of LoftQ. If this is not None, then LoftQ will be used to quantize the backbone weights
and initialize Lora layers. Also pass `init_lora_weights='loftq'`. Note that you should not pass a
quantized model in this case, as LoftQ will quantize the model itself.
eva_config (`Optional[EvaConfig]`):
The configuration of EVA. At a minimum the dataset argument needs to be set (use the same dataset as for
finetuning).
use_dora (`bool`):
Enable 'Weight-Decomposed Low-Rank Adaptation' (DoRA). This technique decomposes the updates of the weights
into two parts, magnitude and direction. Direction is handled by normal LoRA, whereas the magnitude is
handled by a separate learnable parameter. This can improve the performance of LoRA especially at low
ranks. Right now, DoRA only supports linear and Conv2D layers. DoRA introduces a bigger overhead than pure
LoRA, so it is recommended to merge weights for inference. For more information, see
https://arxiv.org/abs/2402.09353.
layer_replication (`List[Tuple[int, int]]`):
Build a new stack of layers by stacking the original model layers according to the ranges specified. This
allows expanding (or shrinking) the model without duplicating the base model weights. The new layers will
all have separate LoRA adapters attached to them.
runtime_config (`LoraRuntimeConfig`):
Runtime configurations (which are not saved or restored).
lora_bias (`bool`):
Defaults to `False`. Whether to enable the bias term for the LoRA B parameter. Typically, this should be
disabled. The main use case for this is when the LoRA weights were extracted from fully fine-tuned
parameters so the bias of those parameters can be taken into account.
helpbase_model_name_or_path)defaultmetadatazLora attention dimensionNa List of module names or regex expression of the module names to replace with LoRA.For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'.This can also be a wildcard 'all-linear' which matches all linear/Conv1D layers except the output layer.If not specified, modules will be chosen according to the model architecture, If the architecture is not known, an error will be raised -- in this case, you should specify the target modules manually.zRList of module names or regex expression of the module names to exclude from Lora.z
Lora alphag z Lora dropoutFzMSet this to True if the layer to replace stores weight like (fan_in, fan_out)nonez7Bias type for Lora. Can be 'none', 'all' or 'lora_only'a When set to True, uses Rank-Stabilized LoRA which sets the adapter scaling factor to `lora_alpha/math.sqrt(r)`, since it was proven to work better. Otherwise, it will use the original default value of `lora_alpha/r`.a List of modules apart from LoRA layers to be set as trainable and saved in the final checkpoint. For example, in Sequence Classification or Token Classification tasks, the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved.Ta' How to initialize the weights of the LoRA layers. Passing `'True'` (default) results in the default initialization from the reference implementation from Microsoft. Passing `'gaussian'` results in Gaussian initialization scaled by the LoRA rank for linear and layers. Setting the initialization to `'False'` leads to completely random initialization and *is discouraged.*Pass `'eva'` results in a data-driven initialization of Explained Variance Adaptation.Passing `'olora'` results in OLoRA initialization.Passing `'pissa'` results in PiSSA initialization.Passing `'pissa_niter_[number of iters]'` initiates Fast-SVD-based PiSSA initialization, where [number of iters] indicates the number of subspace iterations to perform fsvd, and must be a nonnegative integer.Pass `'loftq'` to use LoftQ initializationa The layer indexes to transform, is this argument is specified, PEFT will transform only the layers indexes that are specified inside this list. If a single integer is passed, PEFT will transform only the layer at this index. This only works when target_modules is a list of str.a" The layer pattern name, used only if `layers_to_transform` is different to None and if the layer pattern is not in the common layers pattern.This only works when target_modules is a list of str. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`.zThe mapping from layer names or regexp expression to ranks which are different from the default rank specified by `r`. For example, `{model.decoder.layers.0.encoder_attn.k_proj: 8`})default_factoryr zThe mapping from layer names or regexp expression to alphas which are different from the default alpha specified by `lora_alpha`. For example, `{model.decoder.layers.0.encoder_attn.k_proj: 32`}a The TransformerConfig from Megatron. It is used to create LoRA's parallel linear layer.You can get it like this, `core_transformer_config_from_args(get_args())`, these two functions being from Megatron.You need to specify this parameter when you want to apply LoRA to the ColumnParallelLinear and RowParallelLinear layers of megatron.It should be noted that we may not be able to use the `save_pretrained` and `from_pretrained` functions, because TransformerConfig may not necessarily be serialized.But when using megatron, we can use `get_peft_model_state_dict` function and megatron's framework, they can also save and load models and configurations.z
megatron.corezThe core module from Megatron, it is used to create LoRA's parallel linear layer. It only needs to be passed in when you need to use your own modified megatron core module. Otherwise, it will use the default value `megatron.core`. zThe configuration of LoftQ. If this is passed, then LoftQ will be used to quantize the backbone weights and initialize Lora layers. Also set `init_lora_weights='loftq'` in this case.zThe configuration of EVA. If this is passed, then EVA will be used to intialize the LoRA layers. Also set `init_lora_weights='eva'` in this case. a
Enable 'Weight-Decomposed Low-Rank Adaptation' (DoRA). This technique decomposes the updates of the weights into two parts, magnitude and direction. Direction is handled by normal LoRA, whereas the magnitude is handled by a separate learnable parameter. This can improve the performance of LoRA, especially at low ranks. Right now, DoRA only supports linear and Conv2D layers. DoRA introduces a biggeroverhead than pure LoRA, so it is recommended to merge weights for inference.a( This enables using LoRA to effectively expand a transformer model to a larger size by repeating some layers. The transformation handles models (currently Llama, Bert or Falcon compatible architectures) with a module list in the model which it modifies to expand the number of modules. Base weights are shared so the memory usage is close to the original model. The intended use is these base weights remain fixed during finetuning but each layer has a separate LoRA adapter so the layers can be specialed via the adapter layers fit during fine tuning.The format is a list of [start, end) pairs which specify the layer ranges to stack. For example:
Original model has 5 layers labelled by their position in the model: `[0, 1, 2, 3, 4]`
layer_replication: `[[0, 4], [2, 5]]`
Final model will have this arrangement of original layers: `[0, 1, 2, 3, 2, 3, 4]`
This format is based on what is used for pass-through merges in mergekit. It makes it simple to select sequential ranges of a model and stack them while reusing layers at either end of each sequence.zWhether to enable the bias term for the LoRA B parameter. Typically, this should be disabled. The main use case for this is when the LoRA weights were extracted from fully fine-tuned parameters so the bias of those parameters can be taken into account. CAUSAL_LMzThe type of task.LORAzThe type of PEFT model.zEAn auto mapping dict to help retrieve the base model class if needed.c
\ V 4 p. pVP 4 FG w r4VR/ . 39 d VP V4 K" \ V\ 4 ' g K: \ V4 W& KI V F pVP
V4 K V# )zc
Returns the configuration for your adapter model as a dictionary. Removes runtime configurations.
N)r itemsappend
isinstancer
listpop)selfkv removed_keyskvs & W/home/luca/screen-checking/.venv/lib/python3.14/site-packages/gptqmodel/adapter/peft.pyto_dictLoraConfig.to_dict3 sx
D\ HHJDAT2rN" ##A&As## Q A FF1I c < V ^8 d QhRS[ /# ) save_dirstr)format
__classdict__s "r$ __annotate__LoraConfig.__annotate__H s r' c ^RI Hp \ P RV R24 \ P
! VRR7 \
\ P P W4 RRR7 ;_uu_ 4 pV P 4 p\ P ! V^R 7 p\ P R
V 24 VP V4 R R R 4 R # + ' g i R # ; i) z4Writes the LoRA config in PEFT-compatible JSON form.)HF_ADAPTER_CONFIG_FILE_NAMEz(Adapter: Saving EoRA/Lora config to -> ``T)exist_okwzutf-8)encoding)indentzSaved Adapter Config:
N)
adapter.adapterr2 loginfoosmakedirsopenpathjoinr% jsondumpswrite)r r* r2 fdjson_strs && r$ save_pretrainedLoraConfig.save_pretrainedH s B ;H:QGH
Ht,
"'',,xEsU\
]
]ab Azz!A.H HH/z: ;
GGH ^
]
]
]s
+AC C c N RV n \ V P \ 4 ' d \ V P 4 M V P V n \ V P
\ 4 ' d \ V P
4 M V P
V n \ V P \ 4 ' d V P e \ R4 h\ V P \ 4 ' d V P e \ R4 hV P ' d V P ' g \ R4 hV P ' d V P ' d \ R4 hV P R8X d \ R4 hV P ' d \ R4 hV P R 8X d V P f \ R
4 hV P R 8w d V P e \ R
4 hV P ' d \ R 4 hV P ' d V P" ' g V P$ ' d` \ V P \ 4 ' d" V P P' R 4 ' g V P R
8X d \ R4 hR# R# R# )zONormalizes list inputs and validates the subset of PEFT options supported here.r NzM`Adapter: layers_to_transform` cannot be used when `target_modules` is a str.zHAdapter: `layers_pattern` cannot be used when `target_modules` is a str.z[Adapter: When `layers_pattern` is specified, `layers_to_transform` must also be specified. z Adapter: DoRA is not supportedloftqz Adapter: LoftQ is not supportedevaz Adapter: Eva is not supportedz#Adapter: Lora bias is not supportedpissaoloraz Adapter: RSLora is not supported) peft_typer target_modulesr setexclude_modulesr, layers_to_transform
ValueErrorlayers_patternuse_doramegatron_configinit_lora_weights loftq_config
eva_config lora_bias
use_rslora rank_pattern
alpha_pattern
startswith)r s &r$
__post_init__LoraConfig.__post_init__V s (243F3F(M(MC## $SWSfSf
*4D4H4H$)O)OC$$ %UYUiUi
d))3 / /D4L4L4X lm m d))3 / /D4G4G4S gh h t'?'?'? z{ { ===T 1 1 1 => > ! !W , >? ?
>? ?
#
#u
, 1H <= =
#
#u
, 1L <= = >>> BC C
OOO"""d&8&8&8D 2 2C88d>T>T>_>_`g>h>h**g5 ?@ @ 6 '9
r' c & < V ^8 d QhRS[ RS[ /# )r) r> filenamer+ )r- r. s "r$ r/ r0 s " "3 "# "r' c
\ WR7 p\ VR4 ;_uu_ 4 p\ P ! V4 pRRR4 \ V 4 Uu0 uF qfP
kK ppXP
4 UU u/ uF w rW9 g K
V RR. / R39 g K WbK ppp V ! R/ VB # + ' g i Lt; iu upi u up pi )zCLoads a saved LoRA config and filters out unsupported empty fields.)r> ra rNr )r r= r@ loadr namer )
clsr> ra
resolved_pathfile config_dictr valid_fieldsr" r# s
&&& r$ from_pretrainedLoraConfig.from_pretrained s %$B
-
%
% ))D/K & 17s < u
<(3(9(9(;u(;q?PtUV_cegikmoqs^tUttqt(; u![!! &
% =us# B$
B76
B< B<B<$ B4 c 6 < V ^8 d Qh/ S[ ;R& S[;R& S[S[S[S[ , S[ 3, , ;R& S[S[S[S[ , S[ 3, , ;R& S[;R& S[;R& S[;R& S[R, ;R& S[;R & S[S[S[ , , ;R
& S[S[R, , ;R & S[S[S[S[, S[3, , ;R & S[S[S[S[ , S[ 3, , ;R
& S[S[, ;R& S[S[, ;R& S[S[, ;R& S[S[ , ;R& S[S[ S[3, ;R& S[S[ , ;R& S[;R& S[S[S[
S[S[3, , , ;R& S[;R& S[S[ , ;R& S[S[ , ;R& S[S[, ;R& # ) r) r rc rN rP
lora_alpha lora_dropoutfan_in_fan_outbiasrZ modules_to_saverV rQ rS r[ r\ rU
megatron_corerW rX rT layer_replicationrY task_typerM auto_mapping)r all lora_only)gaussianrJ rL rK z pissa_niter_[number of iters]rI ) r, intr r r floatboolr dictr tuple)r- r. s "r$ r/ r0 s? B !bC F LG H U49c> 23 I ` eDIsN 34 a h Gi j Ok l m t ,
- u z
{ P d3i( Q b
wdeea F "%S 3"78 G T U49c> 23 U d 4. e v D> w H d^ I h C= i ~ T " P
Q b c | U38_ 56 } b c z }a{ | }b} ~ 4. r' )rP rM rN )(__name__
__module__ __qualname____firstlineno____doc__r r rc rN rP ro rp rq rr rZ rs rV rQ rS r~ r[ r\ rU rt rW rX rT ru rY rv rM rw r% rF r^ classmethodrl __annotate_func____static_attributes____classdictcell__)r. s @r$ r r s ^~ $)vG`>a#b
10J'K
LA6; v
7N 8=no8O A 0FGJ v~6NOL ij N 16&*c!d1D ,
J ,1 v
,O =
& a bI#((o p$L* 3Aj " "y r' r )r@ r; dataclassesr r r r typingr r r r
r adapter.remoter utils.loggerr
r9 r rd r' r$ r s@ 8 8 5 5 ) 'n
F" F" F"r'