Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error with workers #6

Closed
nealrs opened this issue Mar 24, 2015 · 12 comments
Closed

error with workers #6

nealrs opened this issue Mar 24, 2015 · 12 comments
Labels
bug

Comments

@nealrs
Copy link

nealrs commented Mar 24, 2015

Hey Jasim, I'm having a little trouble with your script. I've successfully used the text watermarking, but whenever i try to watermark with an image, i get this error (and I do have pillow & colour installed):

$./watermarker ~/desktop/i.jpg ~/desktop/o --wm-image ~/desktop/80.png --output-size 800x600 --output-format png
2015-03-24 11:29:51,543 INFO watermark 87496  Getting list of images: /Users/neal/desktop/i.jpg
2015-03-24 11:29:51,571 INFO watermark 87496  Setting Watermark image: /Users/neal/desktop/80.png
Process PoolWorker-1:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
TypeError: __new__() takes exactly 4 arguments (2 given)
@jasimmk
Copy link
Owner

jasimmk commented Mar 24, 2015

Could you please paste me these below items?

  • ./watermarker ~/Desktop/ ~/desktop/genReport/ --wm-image ~/Desktop/start_wiki.png --output-size 800x600 --output-format png --logging-level DEBUG, debug output
  • also python version you are using? eg: Python 2.7.9 (default, Dec 15 2014, 10:01:34) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin

@nealrs
Copy link
Author

nealrs commented Mar 24, 2015

Jasim,

I'm using 2.7.5

Python 2.7.5 (default, Mar  9 2014, 22:15:05) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin

as for the debug output, here's what i got:

$./watermarker ~/Desktop/ ~/desktop/o/ --wm-image ~/Desktop/i.jpg --output-size 800x600 --output-format png --logging-level DEBUG

2015-03-24 15:25:18,317 INFO watermark 91425  Getting list of images: /Users/neal/Desktop/
2015-03-24 15:25:18,361 INFO watermark 91425  Setting Watermark image: /Users/neal/Desktop/i.jpg
2015-03-24 15:25:18,525 DEBUG watermark.job 91426  Calling: job_function with: args: ('/Users/neal/Desktop/db53.mov_ZoVoTv8i.gif',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3368>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:18,527 DEBUG watermark.workflow 91426  Calling: image_watermark with: args: (<PIL.GifImagePlugin.GifImageFile image mode=P size=640x480 at 0x109DB35A8>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3368>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:18,543 DEBUG watermark.workflow 91426  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3368>,), kwargs:{'percent': 10.0}
2015-03-24 15:25:18,555 CRITICAL watermark.job 91426  bad transparency mask
2015-03-24 15:25:18,635 DEBUG watermark.job 91427  Calling: job_function with: args: ('/Users/neal/Desktop/ts.gif',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB33B0>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:18,638 DEBUG watermark.workflow 91427  Calling: image_watermark with: args: (<PIL.GifImagePlugin.GifImageFile image mode=P size=767x471 at 0x109DB35A8>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB33B0>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:18,639 DEBUG watermark.workflow 91427  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB33B0>,), kwargs:{'percent': 9.8125}
2015-03-24 15:25:18,660 CRITICAL watermark.job 91427  bad transparency mask
2015-03-24 15:25:18,690 DEBUG watermark.job 91428  Calling: job_function with: args: ('/Users/neal/Desktop/ts2.gif',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB33F8>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:18,693 DEBUG watermark.workflow 91428  Calling: image_watermark with: args: (<PIL.GifImagePlugin.GifImageFile image mode=P size=767x471 at 0x109DB35A8>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB33F8>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:18,694 DEBUG watermark.workflow 91428  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB33F8>,), kwargs:{'percent': 9.8125}
2015-03-24 15:25:18,713 CRITICAL watermark.job 91428  bad transparency mask
2015-03-24 15:25:18,756 DEBUG watermark.job 91429  Calling: job_function with: args: ('/Users/neal/Desktop/ts3.gif',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3440>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:18,760 DEBUG watermark.workflow 91429  Calling: image_watermark with: args: (<PIL.GifImagePlugin.GifImageFile image mode=P size=767x471 at 0x109DB35A8>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3440>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:18,760 DEBUG watermark.workflow 91429  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3440>,), kwargs:{'percent': 9.8125}
2015-03-24 15:25:18,775 CRITICAL watermark.job 91429  bad transparency mask
2015-03-24 15:25:18,803 DEBUG watermark.job 91426  Calling: job_function with: args: ('/Users/neal/Desktop/ts5.gif',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3AB8>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:18,807 DEBUG watermark.workflow 91426  Calling: image_watermark with: args: (<PIL.GifImagePlugin.GifImageFile image mode=P size=863x577 at 0x109DB3368>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3AB8>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:18,807 DEBUG watermark.workflow 91426  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3AB8>,), kwargs:{'percent': 12.020833333333334}
2015-03-24 15:25:18,828 CRITICAL watermark.job 91426  bad transparency mask
2015-03-24 15:25:18,835 DEBUG watermark.job 91427  Calling: job_function with: args: ('/Users/neal/Desktop/B1m8ZHrCMAAUaev.jpg',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3A70>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:18,836 DEBUG watermark.workflow 91427  Calling: image_watermark with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=599x444 at 0x109DB33B0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3A70>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:18,837 DEBUG watermark.workflow 91427  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3A70>,), kwargs:{'percent': 9.25}
2015-03-24 15:25:18,896 DEBUG watermark.job 91428  Calling: job_function with: args: ('/Users/neal/Desktop/i.jpg',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3A70>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:18,898 DEBUG watermark.workflow 91428  Calling: image_watermark with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB35A8>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3A70>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:18,898 DEBUG watermark.workflow 91428  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3A70>,), kwargs:{'percent': 12.5}
2015-03-24 15:25:18,901 CRITICAL watermark.job 91427  bad transparency mask
2015-03-24 15:25:18,936 DEBUG watermark.job 91429  Calling: job_function with: args: ('/Users/neal/Desktop/Lemon-Blueberry-Pancakes-Main-640x479.jpg',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3A70>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:18,937 DEBUG watermark.workflow 91429  Calling: image_watermark with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x479 at 0x109DB3440>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3A70>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:18,937 DEBUG watermark.workflow 91429  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3A70>,), kwargs:{'percent': 9.979166666666666}
2015-03-24 15:25:18,971 CRITICAL watermark.job 91428  bad transparency mask
2015-03-24 15:25:18,974 CRITICAL watermark.job 91429  bad transparency mask
2015-03-24 15:25:18,981 DEBUG watermark.job 91426  Calling: job_function with: args: ('/Users/neal/Desktop/100.png',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3B48>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:18,984 DEBUG watermark.workflow 91426  Calling: image_watermark with: args: (<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=100x36 at 0x109DB3AB8>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3B48>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:18,984 DEBUG watermark.workflow 91426  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3B48>,), kwargs:{'percent': 0.75}
2015-03-24 15:25:19,021 DEBUG watermark.job 91427  Calling: job_function with: args: ('/Users/neal/Desktop/80.png',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB39E0>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:19,023 DEBUG watermark.workflow 91427  Calling: image_watermark with: args: (<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=100x36 at 0x109DB3A70>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB39E0>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:19,023 DEBUG watermark.workflow 91427  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB39E0>,), kwargs:{'percent': 0.75}
2015-03-24 15:25:19,024 CRITICAL watermark.job 91427  bad transparency mask
2015-03-24 15:25:19,024 CRITICAL watermark.job 91426  bad transparency mask
2015-03-24 15:25:19,049 DEBUG watermark.job 91428  Calling: job_function with: args: ('/Users/neal/Desktop/Screen Shot 2014-07-09 at 11.00.13 PM.png',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3B90>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:19,051 DEBUG watermark.workflow 91428  Calling: image_watermark with: args: (<PIL.PngImagePlugin.PngImageFile image mode=RGB size=985x119 at 0x109DB3A70>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3B90>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:19,052 DEBUG watermark.workflow 91428  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3B90>,), kwargs:{'percent': 2.4791666666666665}
2015-03-24 15:25:19,071 CRITICAL watermark.job 91428  bad transparency mask
2015-03-24 15:25:19,078 DEBUG watermark.job 91429  Calling: job_function with: args: ('/Users/neal/Desktop/Screen Shot 2015-03-18 at 8.57.57 PM.png',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3B00>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:19,081 DEBUG watermark.workflow 91429  Calling: image_watermark with: args: (<PIL.PngImagePlugin.PngImageFile image mode=RGB size=942x765 at 0x109DB3A70>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3B00>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:19,082 DEBUG watermark.workflow 91429  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3B00>,), kwargs:{'percent': 15.937499999999998}
2015-03-24 15:25:19,161 DEBUG watermark.job 91427  Calling: job_function with: args: ('/Users/neal/Desktop/xlarge.png',), kwargs:{'wm_img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3878>, 'output_format': 'png', 'wm_position': 'BOTTOM_RIGHT', 'output_size': (800, 600), 'output_dir': '/Users/neal/desktop/o/'}
2015-03-24 15:25:19,162 DEBUG watermark.workflow 91427  Calling: image_watermark with: args: (<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=695x494 at 0x109DB39E0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3878>), kwargs:{'posx': 'RIGHT', 'posy': 'BOTTOM'}
2015-03-24 15:25:19,163 DEBUG watermark.workflow 91427  Calling: resize with: args: (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=800x600 at 0x109DB3878>,), kwargs:{'percent': 10.291666666666668}
2015-03-24 15:25:19,175 CRITICAL watermark.job 91429  bad transparency mask
2015-03-24 15:25:19,202 CRITICAL watermark.job 91427  bad transparency mask

if i instead run the conversion on a png i get a similar trace, but it ends with this recursive/endless KeyboardInterrupt that I have to kill-9.

@jasimmk
Copy link
Owner

jasimmk commented Mar 25, 2015

Thanks @nealrs, Could reproduce the issue with non transparent watermark images. Will be coming up with fix asap

jasimmk added a commit that referenced this issue Mar 25, 2015
@jasimmk
#6 Fix for bad transparency mask
da89eb4
@jasimmk
Copy link
Owner

jasimmk commented Mar 25, 2015

From PIL's documentation


    def paste(self, im, box=None, mask=None):
        ....
        If a mask is given, this method updates only the regions
        indicated by the mask.  You can use either "1", "L" or "RGBA"
        images (in the latter case, the alpha band is used as mask).
        Where the mask is 255, the given image is copied as is.  Where
        the mask is 0, the current value is preserved.  Intermediate
        values can be used for transparency effects.
        .....

Its clear that, if we use a mask, it should be in mode "1", "L" or "RGBA"
So changing the image_watermark_method to use mask only if the watermark is

Could you please checkout the branch jas/6/bad-transparency-mark and verify whether issue still exist?

@nealrs
Copy link
Author

nealrs commented Mar 25, 2015

Jasim,

I can embed non-transparent background images & text, but I'm still getting this with the new branch when I try a transparent image:

./watermarker ~/desktop/i.jpg ~/desktop/o --wm-image ~/desktop/80.png --output-size 800x600 --output-format png
2015-03-25 13:20:06,175 INFO watermark 97055  Getting list of images: /Users/neal/desktop/i.jpg
2015-03-25 13:20:06,210 INFO watermark 97055  Setting Watermark image: /Users/neal/desktop/80.png
Process PoolWorker-1:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
TypeError: __new__() takes exactly 4 arguments (2 given)

@nealrs
Copy link
Author

nealrs commented Mar 25, 2015

i did have success with a non-transparent gif

@jasimmk
Copy link
Owner

jasimmk commented Mar 26, 2015

When I almost try to reproduce the situation, the system works smooth :(

$ ./watermarker ~/Desktop/i.jpg ~/Desktop/o --wm-image ~/Desktop/80.png --output-size 800x600 --output-format png
2015-03-26 07:09:32,411 INFO watermark 2502  Getting list of images: /Users/Jasim/Desktop/i.jpg
2015-03-26 07:09:32,424 INFO watermark 2502  Setting Watermark image: /Users/Jasim/Desktop/80.png
2015-03-26 07:09:32,496 INFO watermark.workflow 2503  Saving output image at: /Users/Jasim/Desktop/o/i.png

Could you please paste a DEBUG log of this error?

@nealrs
Copy link
Author

nealrs commented Mar 26, 2015

Hey Jasim, can you try with this input & watermark image?
watermark: 80

input: i

@jasimmk
Copy link
Owner

jasimmk commented Mar 26, 2015

Strange!!!
Yes I can reproduce the error. Going through

jasimmk added a commit that referenced this issue Mar 28, 2015
@jasimmk
#6 Making sure, serialized data is passed to multiprocessor worker fu…
fc5fafe 
…nctions
jasimmk added a commit that referenced this issue Mar 28, 2015
@jasimmk
Merge pull request #11 from jasimmk/jas/6/multi-proc-errors
c496cdd 
#6 Muliprocessing, Serialization issue
@jasimmk
Copy link
Owner

jasimmk commented Mar 28, 2015

There were some serialization issues with PIL image files, passed to multiprocessing job. Implemented pickle serialization and everything is smooth now

@nealrs
Copy link
Author

nealrs commented Mar 28, 2015

thanks @jasimmk, looks like that fixed the errors -- one more quick question though: how is the watermark sized & is there a ratio/pixel density I should use?

@jasimmk jasimmk added the bug label Mar 29, 2015
@jasimmk
Copy link
Owner

jasimmk commented Mar 29, 2015

Hi @nealrs , Closing this bug now. As of now, the watermark size is calculated based on relative size of common watermarks. If you find it very handy, that we need to control size of watermarks, We could add it as new feature. Could you please comment to this #12 [Command line argument to control watermark size] ticket?.

@jasimmk jasimmk closed this as completed Mar 29, 2015
@nealrs nealrs mentioned this issue Mar 29, 2015
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jasimmk @nealrs