How are multicategorical features processed?

[davidfstein]

How are multi-categorical features processed?

[akihironitta]

I'd suggest looking at the implementation and its test case as we have no docs around this yet:

pytorch-frame/torch_frame/data/mapper.py

Lines 118 to 190 in 3710d2f

	class MultiCategoricalTensorMapper(TensorMapper):
	r"""Maps any multi-categorical series into an index representation, with
	:obj:`-1` denoting missing values (NaN) and no value denoting not belonging
	to any categories. Unseen categories will be ignored.
	Args:
	categories (List[Any]): A list of possible categories in the
	multi-categorical column sorted by counts.
	sep (str, optional): The delimiter for the categories in each cell.
	If :obj:`None`, it assumes each cell is directly represented as a
	list of categories. (default: :obj:`None`)
	"""
	def __init__(
	self,
	categories: list[Any],
	sep: None | str = None,
	):
	super().__init__()
	self.categories = categories
	self.sep = sep
	self.index = pd.Series(
	index=categories,
	data=pd.RangeIndex(0, len(categories)),
	)
	self.index = pd.concat((self.index, (pd.Series([-1], index=[-1]))))
	@staticmethod
	def split_by_sep(row: str | Iterable | None, sep: None | str) -> set[Any]:
	if row is None or row is np.nan:
	return {-1}
	elif isinstance(row, str):
	assert sep is not None
	if row.strip() == '':
	return set()
	else:
	return {cat.strip() for cat in row.split(sep)}
	elif isinstance(row, Iterable):
	assert sep is None
	return set(row)
	else:
	raise ValueError(
	f"MulticategoricalTensorMapper only supports str or list "
	f"types (got input: {row})")
	def forward(
	self,
	ser: Series,
	*,
	device: torch.device | None = None,
	) -> MultiNestedTensor:
	if ser.dtype != 'object':
	raise ValueError('Multi-categorical types expect string as input')
	values = []
	original_index = ser.index
	ser = ser.apply(lambda row: MultiCategoricalTensorMapper.split_by_sep(
	row, sep=self.sep))
	ser = ser.explode()
	ser = pd.merge(
	ser.rename('data'),
	self.index.rename('index'),
	how='left',
	left_on='data',
	right_index=True,
	).dropna()
	ser['index'] = ser['index'].astype('int64')
	values = torch.from_numpy(ser['index'].values)
	offset = ser.index.value_counts()
	offset = offset.reindex(original_index, fill_value=0)
	offset = pd.concat((pd.Series([0]), offset))
	offset = torch.from_numpy(offset.values)
	offset = torch.cumsum(offset, dim=0)
	return MultiNestedTensor(num_rows=len(original_index), num_cols=1,
	values=values, offset=offset)

pytorch-frame/test/data/test_mapper.py

Lines 64 to 91 in 3710d2f

	def test_multicategorical_tensor_mapper():
	for ser in [
	pd.Series(['A,B', 'B', '', 'C', 'B,C', None, np.nan]),
	# Testing with leading and traling whitespace
	pd.Series([' A, B', ' B', ' ', 'C ', 'B , C', None, np.nan])
	]:
	expected_values = torch.tensor([1, 0, 0, 0, -1, -1])
	expected_boundaries = torch.tensor([0, 2, 3, 3, 3, 4, 5, 6])
	mapper = MultiCategoricalTensorMapper(['B', 'A'], sep=",")
	tensor = mapper.forward(ser)
	values = tensor.values
	offset = tensor.offset
	assert values.dtype == torch.long
	assert torch.equal(
	values[expected_boundaries[0]:expected_boundaries[1]].sort().
	values, torch.tensor([0, 1]))
	assert torch.equal(values[expected_boundaries[1]:],
	expected_values[expected_boundaries[1]:])
	assert torch.equal(offset, expected_boundaries)
	out = mapper.backward(tensor)
	assert out.values[0] == 'A,B' or out.values[0] == 'B,A'
	assert out.values[1] == 'B'
	assert out.values[2] == ''
	assert out.values[3] == ''
	assert out.values[4] == 'B'
	assert out.values[5] == ''

[akihironitta]

Let me know here if you have any further questions about this!

[davidfstein]

So in this example, index of B=0 and index of A=1. So 'A,B' = 1 + 0 = 1 and 'B' = 0. 'C' is not a recognized category so it is also 0? It seems odd to have non-recognized categories be represented with the same value as a recognized category. Am I understanding correctly?

[akihironitta]

'C' is not a recognized category so it is also 0?

In short, yes.

Currently, multi-categorical stype columns support NAStrategy.ZEROS as in:

pytorch-frame/torch_frame/typing.py

Lines 92 to 94 in 3710d2f

	@property
	def is_multicategorical_strategy(self) -> bool:
	return self == NAStrategy.ZEROS

and it means that we always map unknown categories to 0 (e.g., 'C' to 0):

pytorch-frame/torch_frame/nn/encoder/stype_encoder.py

Lines 120 to 121 in 3710d2f

	elif self.na_strategy == NAStrategy.ZEROS:
	fill_value = 0

Mapping unknown categories to 0 means that we map them to the most frequent category as we sort category counts, and we expect 0 is the most frequent category:

pytorch-frame/torch_frame/data/stats.py

Line 120 in 3710d2f

count = ser.value_counts(ascending=False)

We definitely should have a better explaination around this in our docs, but thanks for asking here!

[davidfstein]

Ahhh that makes sense! Got it, thank you