[IconFetcher] Don't free on removal from thread-pool #539

Job	Run time
build-autotools-gcc-no-window	4m 15s
build-autotools-gcc	2m 59s
build-autotools-clang	3m 6s
build-meson-clang	1m 54s
build-meson-gcc	1m 48s
	14m 2s

Provide feedback