Hugging Face - Transformers

Model

Training

Tokenizer

  • tokenizers.BertWordPieceTokenizer - for vocab generation - impl

  • tokenizers.normalizers.BertNormalizer - impl

  • transformers.tokenization_bert.BertTokenizer - tokenizer for normal usage - impl

  • transformers.tokenization_bert.BertTokenizerFast - fast tokenizer for normal usage - impl

  • tokenizers.AutoTokenizer - doc - impl

  • PreTrainedTokenizerBase.__call__ - doc

Pipelines

Data Handling

Important Torch Classes