Class TokensInfo (1.144.0)

1.144.0 (latest) 1.143.0 1.142.0 1.141.0 1.140.0 1.139.0 1.138.0 1.137.0 1.136.0 1.135.0 1.134.0 1.133.0 1.132.0 1.131.0 1.130.0 1.129.0 1.122.0 1.121.0 1.120.0 1.119.0 1.118.0 1.117.0 1.116.0 1.115.0 1.114.0 1.113.0 1.112.0 1.111.0 1.110.0 1.109.0 1.108.0 1.107.0 1.106.0 1.105.0 1.104.0 1.103.0 1.102.0 1.101.0 1.100.0 1.99.0 1.98.0 1.97.0 1.96.0 1.95.1 1.94.0 1.93.1 1.92.0 1.91.0 1.90.0 1.89.0 1.88.0 1.87.0 1.86.0 1.85.0 1.84.0 1.83.0 1.82.0 1.81.0 1.80.0 1.79.0 1.78.0 1.77.0 1.76.0 1.75.0 1.74.0 1.73.0 1.72.0 1.71.1 1.70.0 1.69.0 1.68.0 1.67.1 1.66.0 1.65.0 1.63.0 1.62.0 1.60.0 1.59.0 1.58.0 1.57.0 1.56.0 1.55.0 1.54.1 1.53.0 1.52.0 1.51.0 1.50.0 1.49.0 1.48.0 1.47.0 1.46.0 1.45.0 1.44.0 1.43.0 1.39.0 1.38.1 1.37.0 1.36.4 1.35.0 1.34.0 1.33.1 1.32.0 1.31.1 1.30.1 1.29.0 1.28.1 1.27.1 1.26.1 1.25.0 1.24.1 1.23.0 1.22.1 1.21.0 1.20.0 1.19.1 1.18.3 1.17.1 1.16.1 1.15.1 1.14.0 1.13.1 1.12.1 1.11.0 1.10.0 1.9.0 1.8.1 1.7.1 1.6.2 1.5.0 1.4.3 1.3.0 1.2.0 1.1.1 1.0.1 0.9.0 0.8.0 0.7.1 0.6.0 0.5.1 0.4.0 0.3.1

TokensInfo(mapping=None, *, ignore_unknown_fields=False, **kwargs)

Tokens info with a list of tokens and the corresponding list of token ids.

Attributes
Name	Description
tokens	MutableSequence[bytes] A list of tokens from the input.
token_ids	MutableSequence[int] A list of token ids from the input.
role	str Optional. Optional fields for the role from the corresponding Content.

Methods

TokensInfo

TokensInfo(mapping=None, *, ignore_unknown_fields=False, **kwargs)

Tokens info with a list of tokens and the corresponding list of token ids.