Degrade from 3.4 to 3.5: Windows Protobuf Compiler does no longer understand * operator for multiple input files

[apacha]

I tried to follow the instructions from tensorflow-models repo up until the point where to run

protoc object_detection/protos/*.proto --python_out=.

So I downloaded the appropriate Windows version and added it to my Path. But when running the corresponding windows command:

protoc .\object_detection\protos\*.proto --python_out=.
or even with the forward slash
protoc object_detection/protos/*.proto --python_out=.

I only get the error object_detection/protos/*.proto: No such file or directory, indicating that it does not understand the *-operator. When compiling the files individually that reside in that directory, it works as expected.

[liujisi]

@laszlocsomor I suspect this has something to do with the long path support.

[laszlocsomor]

I suspect this never worked on Windows.

On Linux, Bash expands the wildcards so the main method's argv already contains the list of files, not the wildcard.

[apacha]

@laszlocsomor Very likely, this behavior never worked, but it would be nice, if it did work.

[laszlocsomor]

I have a fix for it. Stay tuned.

[Aisuko]

I have the same issue on my win10 and anaconda3 , with the instructions from tensorflow-models repo up until the point where to run

 protoc object_detection/protos/*.proto --python_out=.

[apacha]

There is a nice script here that you can use instead: tensorflow/models#1934

[Aisuko]

Thanks 😘 @apacha
This my solution:

Head to the protoc releases page and download the protoc-3.4.0-win32.zip, extract it, and you will find protoc.exe in the bin directory.

Move this to models/research folder clone to local tensorflow model project

Next,you can use the protoc command like

protoc object_detection/protos/*.proto --python_out=.

Reference linking

[Aisuko]

@apacha I think that i have been succeed.

[apacha]

@Aisuko You're right. This actually works in Version 3.4.0 of protoc but not with version 3.5.0 and is therefore a degrade of the Windows protoc.exe.

[acozzette]

@pherl What's the current status of this issue? Is this now fixed on master?

[liujisi]

My understanding we probably would have to build the protoc directly using mingw in windows without using cross-compile. I will see if we can fix this in 3.6

[jeffreytang]

Just tried latest 3.6.0 on Windows 10 command window, still could not recognize "*.proto"

[vthoppae]

Using batch file try this option using for loop command
for /f %%G IN ('dir /b "C:\Protobuf\protoc-3.6.0\in*.proto"') do (
protoc --proto_path=C:\Protobuf\protoc-3.6.0\in --java_out=C:\Protobuf\protoc-3.6.0\out %%G
)

[moshebitan]

This is one liner to replace google command for compliing proto files.
ORIGINAl (Doesn't work on windows):
protoc object_detection\protos\*.proto --python_out=.

Batch version works on windows:
for /f %G in ('dir /b object_detection\protos\*.proto') do protoc object_detection\protos\%G --python_out=.

[laszlocsomor]

I fixed this, and it works with long paths too:

1. In MSYS:

   $ find /c/tmp/proto/aaaaaaaa.aaa/ -name '*.proto' -type f
   /c/tmp/proto/aaaaaaaa.aaa/bbbbbbbb.bbb/cccccccc.ccc/dddddddd.ddd/eeeeeeee.eee/ffffffff.fff/gggggggg.ggg/hhhhhhhh.hhh/iiiiiiii.iii/jjjjjjjj.jjj/kkkkkkkk.kkk/llllllll.lll/mmmmmmmm.mmm/nnnnnnnn.nnn/oooooooo.ooo/pppppppp.ppp/qqqqqqqq.qqq/rrrrrrrr.rrr/ssssssss.sss/tttttttt.ttt/uuuuuuuu.uuu/vvvvvvvv.vvv/wwwwwwww.www/a.proto
   /c/tmp/proto/aaaaaaaa.aaa/bbbbbbbb.bbb/cccccccc.ccc/dddddddd.ddd/eeeeeeee.eee/ffffffff.fff/gggggggg.ggg/hhhhhhhh.hhh/iiiiiiii.iii/jjjjjjjj.jjj/kkkkkkkk.kkk/llllllll.lll/mmmmmmmm.mmm/nnnnnnnn.nnn/oooooooo.ooo/pppppppp.ppp/qqqqqqqq.qqq/rrrrrrrr.rrr/ssssssss.sss/tttttttt.ttt/uuuuuuuu.uuu/vvvvvvvv.vvv/wwwwwwww.www/b.proto
   
   $ find /c/tmp/proto/aaaaaaaa.aaa/ -name '*.h' -type f | wc -l
   0
   

2. In cmd.exe:

   c:\>cd c:\tmp\proto
   
   c:\tmp\proto>protoc-dev.exe aaaaaaaa.aaa\bbbbbbbb.bbb\cccccccc.ccc\dddddddd.ddd\eeeeeeee.eee\ffffffff.fff\gggggggg.ggg\hhhhhhhh.hhh\iiiiiiii.iii\jjjjjjjj.jjj\kkkkkkkk.kkk\llllllll.lll\mmmmmmmm.mmm\nnnnnnnn.nnn\oooooooo.ooo\pppppppp.ppp\qqqqqqqq.qqq\rrrrrrrr.rrr\ssssssss.sss\tttttttt.ttt\uuuuuuuu.uuu\vvvvvvvv.vvv\wwwwwwww.www\*.proto --cpp_out=.
   

3. In MSYS:

   $ find /c/tmp/proto/aaaaaaaa.aaa/ -name '*.h' -type f
   /c/tmp/proto/aaaaaaaa.aaa/bbbbbbbb.bbb/cccccccc.ccc/dddddddd.ddd/eeeeeeee.eee/ffffffff.fff/gggggggg.ggg/hhhhhhhh.hhh/iiiiiiii.iii/jjjjjjjj.jjj/kkkkkkkk.kkk/llllllll.lll/mmmmmmmm.mmm/nnnnnnnn.nnn/oooooooo.ooo/pppppppp.ppp/qqqqqqqq.qqq/rrrrrrrr.rrr/ssssssss.sss/tttttttt.ttt/uuuuuuuu.uuu/vvvvvvvv.vvv/wwwwwwww.www/a.pb.h
   /c/tmp/proto/aaaaaaaa.aaa/bbbbbbbb.bbb/cccccccc.ccc/dddddddd.ddd/eeeeeeee.eee/ffffffff.fff/gggggggg.ggg/hhhhhhhh.hhh/iiiiiiii.iii/jjjjjjjj.jjj/kkkkkkkk.kkk/llllllll.lll/mmmmmmmm.mmm/nnnnnnnn.nnn/oooooooo.ooo/pppppppp.ppp/qqqqqqqq.qqq/rrrrrrrr.rrr/ssssssss.sss/tttttttt.ttt/uuuuuuuu.uuu/vvvvvvvv.vvv/wwwwwwww.www/b.pb.h
   

[Igneous01]

any updates? Will the fix be merged in at some point?

[liyangyang13]

I encountered this problem recently, but unfortunately, this bug is still not fixed in the recent release of Protobuf.
The easiest way to solve this instead of manually fixing is to download the old version protoc-3.4.0-win32https://github.com/protocolbuffers/protobuf/releases?after=v3.4.1 and then following the usual steps. It worked for me.

[OsamaNaeem1995]

Thanks 😘 @apacha
This my solution:

Head to the protoc releases page and download the protoc-3.4.0-win32.zip, extract it, and you will find protoc.exe in the bin directory.

Move this to models/research folder clone to local tensorflow model project

Next,you can use the protoc command like

protoc object_detection/protos/*.proto --python_out=.

Reference linking

i'm still getting the same error ,
**no such file or directory **

Thanks 😘 @apacha
This my solution:

Head to the protoc releases page and download the protoc-3.4.0-win32.zip, extract it, and you will find protoc.exe in the bin directory.

Move this to models/research folder clone to local tensorflow model project

Next,you can use the protoc command like

protoc object_detection/protos/*.proto --python_out=.

Reference linking

i am still getting the error despite following the steps you mentioned

[xycui]

Seems that it is keeping break since 3.5.0. And this regression bug has already kept for almost 2 years.

[laszlocsomor]

@xycui : You're right. Sorry for the long time to fix. I'm not on the protobuf team so this was low priority for me.

[xycui]

@laszlocsomor Thank you so much for the help. But it seems that the fixing PR is still pending because of the 'release tag' missing.

[laszlocsomor]

You're welcome. I asked the reviewer to add it.

[apacha]

Halleluja!

[JamesOldfield]

Just for the record: this fix was first released in protobuf 3.10. I haven't properly checked but the commit 5b6238e can be found in the 3.10.x branch.