---
title: "Tokenizer animated icon"
description: "Tokenizer animated icon (Model training & fine-tuning). The word is cut into tokens. Light: two cuts draw down through the word's boundaries, and as they fade the three cells spring apart into separate tokens and stop dead. Strong: the left cut lands, then the right, and the cells spring apart with a small overshoot and settle. The word line never moves; the tokens stay separate."
canonical: https://generativeicons.com/icons/tokenizer/
---

# Tokenizer animated icon

The word is cut into tokens. Light: two cuts draw down through the word's boundaries, and as they fade the three cells spring apart into separate tokens and stop dead. Strong: the left cut lands, then the right, and the cells spring apart with a small overshoot and settle. The word line never moves; the tokens stay separate.

- Id: `tokenizer`
- Category: Model training & fine-tuning
- Search terms: tokenization, tokenisation, tokeniser, tokens, bpe, byte pair encoding, subword, wordpiece, sentencepiece, split text, text segmentation, vocabulary, vocab
- Two states: joined-word to separate-tokens, played forward and back
- Styles: outline, solid, duotone; outline weights 1.5, 2, 2.5 (any from 1 to 3); motion light or strong
- File: https://generativeicons.com/assets/tokenizer.lottie (dotLottie, 7599 bytes, sha256 65b7977db7c39873437b28252232b72d31a53f839b6cc0b8480b77a51d9cf58f)
- License: MIT

## Install

```sh
npm install @lottiefiles/generative-icons-runtime
npx generative-icons init
npx generative-icons add tokenizer
```

## React

```tsx
// npm install @lottiefiles/generative-icons-react
'use client';
import { AnimatedIcon } from '@lottiefiles/generative-icons-react';
import icon, { src } from './generative-icons/tokenizer.mjs';

export function TokenizerIcon() {
  return <AnimatedIcon icon={icon} src={src} size={32}
    color="#ffffff" variant="outline" weight={2}
    motion="light" theme="dark" />;
}
```

## JavaScript

```js
import { createIcon } from '@lottiefiles/generative-icons-runtime';
import icon, { src } from './generative-icons/tokenizer.mjs';

const handle = await createIcon(canvas, {
  icon, src,
  color: '#ffffff', style: 'outline', weight: 2,
  motion: 'light', theme: 'dark'
});
button.addEventListener('click', () => handle.play());
// On removal: handle.destroy();
```

More: https://generativeicons.com/icons/tokenizer/ · https://generativeicons.com/api/icons/tokenizer