Package publish rejects the package

[mcspr]

Configuration

Operating system:

Windows 20H2

PlatformIO Version (platformio --version):

PlatformIO Core, version 5.1.0

Description of problem

Following up on the #3776

After downloading release files from:
https://github.com/earlephilhower/esp-quick-toolchain/releases

> dir | foreach {echo $_.Name}
aarch64-linux-gnu.xtensa-lx106-elf-48f7b08.210128.tar.gz
x86_64-apple-darwin14.xtensa-lx106-elf-48f7b08.210128.tar.gz
x86_64-linux-gnu.xtensa-lx106-elf-48f7b08.210128.tar.gz
x86_64-w64-mingw32.xtensa-lx106-elf-48f7b08.210128.zip
> dir | foreach {pio package publish $_.Name}
Error: Invalid request data for pkgtype -> 'Must be one of: library, platform, tool'
Error: Invalid request data for pkgtype -> 'Must be one of: library, platform, tool'

Aborted!
>

I have not tried to 'repackage', like before, since I assumed this method of directly uploading .tar.gz or .zip should work now.

Additional info

package.json is generated here:

https://github.com/earlephilhower/esp-quick-toolchain/blob/48f7b08d1679365ea462675dd3bfb9d83c9bf11a/Makefile#L280-L290

[maxgerhardt]

Judging from

platformio-core/platformio/clients/registry.py

Lines 44 to 46 in c623a6a

	return self.send_auth_request(
	"post",
	"/v3/packages/%s/%s" % (owner, PackageType.from_archive(archive_path)),

and

platformio-core/platformio/package/meta.py

Lines 54 to 66 in c623a6a

	@classmethod
	def from_archive(cls, path):
	assert path.endswith("tar.gz")
	manifest_map = cls.get_manifest_map()
	with tarfile.open(path, mode="r:gz") as tf:
	for t in sorted(cls.items().values()):
	for manifest in manifest_map[t]:
	try:
	if tf.getmember(manifest):
	return t
	except KeyError:
	pass
	return None

and

platformio-core/platformio/package/manifest/parser.py

Lines 34 to 39 in c623a6a

	class ManifestFileType(object):
	PLATFORM_JSON = "platform.json"
	LIBRARY_JSON = "library.json"
	LIBRARY_PROPERTIES = "library.properties"
	MODULE_JSON = "module.json"
	PACKAGE_JSON = "package.json"

The logic whereit fails is that it can't decide whether it's a library, platform, or tool. And that is decided by the package having a package.json (--> tool), platform.json (--> platform) or library.json/library.properties/module.json (--> library)

The e.g. https://github.com/earlephilhower/esp-quick-toolchain/releases/download/3.0.0-newlib4.0.0-gnu22/x86_64-linux-gnu.xtensa-lx106-elf-48f7b08.210128.tar.gz file has

.. as a direct first thing a folder, not the files directly. Only in there

Is the file we're looking for.

So I think the bug here is that the archive file contains a folder which contains all the files, but it should contain all the files directly. Bug in the generation logic.

[mcspr]

Afaik, archive structure is made to be compatible with ArduinoIDE's downloader, as these archives are also used as-is here when tagged version is released:
https://github.com/esp8266/Arduino/blob/master/package/package_esp8266com_index.template.json
(as development version needs manual sync through the use of tools/get.py)

So, the gist is:

• PackageType.from_archive is not raising exception and is not checked by the caller whether it is None
• `"/v3/packages/mcspr/None" URL is valid on the server side?
• from_archive should be able to determine package root like it already does with platform_packages = $URL installation?
  

  platformio-core/platformio/package/manager/base.py

  Lines 128 to 134 in 6985945

  	def find_pkg_root(self, path, spec): # pylint: disable=unused-argument
  	if self.manifest_exists(path):
  	return path
  	for root, _, _ in os.walk(path):
  	if self.manifest_exists(root):
  	return root
  	raise MissingPackageManifestError(", ".join(self.manifest_names))

[maxgerhardt]

Oh well handy that such a find_pkg_root function already exists, then I'm sure it can be integrated here rather easily (if the bug is as I'm seeing it that it doesn't walk the directory structure). CC @ivankravets

[ivankravets]

So, and where is the issue? I'm not sure that I understand what is going wrong. There is PlatformIo Registry with its own rules, and there 3rd party archives that are not compatible. Is it correct?

[mcspr]

Package archive above do not contain package.json in the 'root' of the archive, but pio package publish still tries to upload it without raising an error until the very last moment. The very last message describes just that - allow it to discover the root, just like the normal installation does when we do pio package pack

[ivankravets]

This is normal behavior. It is a popular practice to put files in the interim folder to avoid issues with ZIP archive that has issues with putting files in the root. It is common practice to ship libraries in ZIP archives and put library.json in the "nested" folder.

[mcspr]

...so shouldn't we be able to publish an archive with <root>/somedir/package.json?

Hence the comments about the PackageType.from_archive. It only checks for <root>/package.json, thus causing the publish to use /v3/packages/mcspr/None as an URL instead of /v3/packages/mcspr/tool (i.e. see the 'map' just above).

[ivankravets]

Hence the comments about the PackageType.from_archive. It only checks for /package.json

Aha, now I understand. Yes, it seems like an issue. Thanks for the report! We will think about providing lightweight solution for this.