Hugging Face - Transformers

Model

Training

Tokenizer

  • tokenizers.BertWordPieceTokenizer - for vocab generation - impl
  • tokenizers.normalizers.BertNormalizer - impl
  • transformers.tokenization_bert.BertTokenizer - tokenizer for normal usage - impl
  • transformers.tokenization_bert.BertTokenizerFast - fast tokenizer for normal usage - impl
  • tokenizers.AutoTokenizer - doc - impl
  • PreTrainedTokenizerBase.__call__ - doc

Pipelines

Data Handling

Important Torch Classes

Last modified July 16, 2022: fix headlines in ML doc (a533e7c)