Screenshot not generated

[rlasjunies]

Hi all,

1st of all Great extension!

The screenshots are not generated on my installation. Looking in the code I feel the screenshots are generated calling this url 'http://dewey-server.azurewebsites.net/screenshot?query='

Unfortunatelly when calling it, I've got the answer 'Error 403 - web app is stopped'

Did I missed something?

[outcoldman]

@artemgrygor could you take a look?

[artemgrygor]

@rlasjunies @outcoldman it was hosted not on my primary subscription and for some reason it was disabled. Will try to setup on the main one with the same url.

[artemgrygor]

@outcoldman I can't setup a new instance with the same url because the previous subscription is not canceled. I've written a support ticket.
I can create a new server on my Bizspark subscription but with the different url only. So we will have to update dewey app itself.
Also @jamiewilson page2image doesn't work locally. Maybe it is a good idea to re-write application, at least, to work with "not show thumbnail" settings. Because right now it doesn't work too due to color-thief which is waiting for a screenshot to grab its main color.
P.S. I will try to play with this one https://github.com/brenden/node-webshot also

[artemgrygor]

ok, thanks to MS support I recreated the subscription. So dewey-server is working now. But the screenshots no.

[outcoldman]

@artemgrygor thank you for doing that! screenshots are always a problem :( @jamiewilson your thoughts?

[rlasjunies]

Thank guys, The 'favicon' are shown in the thumbnail.
Unfortunately not the screenshot (yet? ;-))

[artemgrygor]

@rlasjunies please switch off "Show screenshot" options in the settings menu while we're looking for the solution.

[rlasjunies]

Hi Guys,

I've spent few hours on the webshot. Phantom, used in webshot, seems to be very complex app.
I did stupid browserify trial without success, I've reached a 'process.binding is not supported' error at early stage. I do not know Chrome Ext but I believe we will have issues for the "Node like" process used. For me, it does not smell good ;-(
I would be happy to continue to dive in, but unless someone can show/guide me on this path, I do not think it's a good way to go.

Idea: can chrome ext. open new tabs, on the url we want thumbnail and take screenshot?

PS: I do not know Chrome Ext, Browserify ... so my analysis can be wrong

[rlasjunies]

Hi @artemgrygor, @jamiewilson, @outcoldman

Me again. I did several trialsss on the local screenshot generation.

The only way I found without a dedicated server is:

• add a BrowserAction button in the extension
• on user click, capture the active tab
  • store the screen shot in storage (local only?)
• retrieve the screenshot from this place

Pro:

• no need of dedicated server

Cons:

• require the user action for each screenshot
• store everything on chrome.local storage (in some ways it could be considered as a plus)

I will clean a little bit my trial. If you are interested let me know, I can create a new branch and share with you. Even if it's not as good than the server, one of the main feature of Dewey is still available.

Additionally I've some questions:

• Dewey is an "Chrome App". Is it something mandatory for you, can it be an "Chrome Extension"?
• I would prefer an "Extension" that replace the "new tab" by default. Is it something you are against?
• The query is made on the bookmark, not in the "Chrome app" list. Is it something you do not want?

Richard

[outcoldman]

We thought about generating screenshots on client, but as you see it is not easy and requires additional clicks from user. Plus if computer will be slow (like chromebook) - it can work really bad.

Dewey is an "Chrome App". Is it something mandatory for you, can it be an "Chrome Extension"?

I believe there was a reason for that. Extension could not do something, which we need to do in App.

I would prefer an "Extension" that replace the "new tab" by default. Is it something you are against?

Take a look on https://github.com/deweyapp/deweylauncher

The query is made on the bookmark, not in the "Chrome app" list. Is it something you do not want?

Did not get this question.

[outcoldman]

@artemgrygor have you started to work on integrating node-webshot?

My thoughts about it If we are going to implement our own web server for generating screenshots:

1. We should use https://developer.chrome.com/apps/app_identity to identify all requests to the webserver. This will help us to protect our server from misusing.
2. On web server we should verify authentication token with google servers.
3. We should cache all images and invalidate them not so often.
4. We still should limit number of requests from client (we can actually do that with nginx proxy).

[outcoldman]

@artemgrygor ping

[artemgrygor]

@outcoldman @jamiewilson Omg, sorry guys. For some reason haven't answered this one.
Unfortunately no, I'm quite busy right now. I have no experience with google identity will check what we can do. I agree with all these statements about caching. And also need to check where to store images. Should it be azure or amazon. On azure I have bizpark so we could use this one.

[outcoldman]

@artemgrygor np, I can take a look into it, when I will have a time, don't want us both to work on the same thing. Just wanted to check if you have started looking into it. I will try to build some prototype.

[zeckdude]

I get the following JSON Response when trying to hit http://dewey-server.azurewebsites.net/screenshot?query=https%3A%2F%2Fwebflow.com%2F as an example: {"status":"error","msg":"Call limit for today"}

That's happening with all my screenshots.

[outcoldman]

@artemgrygor do you have any stats - how many requests we were generating for the dewey users in hour / day? Also maybe how much traffic have we used on our proxy server?

[artemgrygor]

@outcoldman yes, I could probably get it from Azure stat but right now the portal is down :-) I can see web apps but not information about app itselft. I got lots of "Failed to load resource: net::ERR_CONNECTION_TIMED_OUT"
Will check in few minutes.

[artemgrygor]

Ok, I got it. There were 458k requests with max 6k requests at one time in the last week.
Important: this info with favicon too. So probably the screenshot numbers should be twice less.

[outcoldman]

458k in a week, right? 6k in an hour?
About 2 requests per second. And I would assume it is about 50Gb in a week traffic, meaning 200Gb in a month. I guess simple droplet on DO can handle it. But that will cost $5 / month.

[artemgrygor]

Yes, looks like correct.
Maybe free azure?

[outcoldman]

@artemgrygor up to you, as far as I know - Azure is not free :D But if you will host it with your account - that will be wonderful.

[artemgrygor]

@outcoldman right now we are using my free subscription, I don't pay for this. We can increase the plan and move it to my bizspark subscription if needed. Will have a look on DO too.

[outcoldman]

@artemgrygor so this is a plan I see:

1. We will use https://developer.chrome.com/apps/app_identity to get the Google Auth Token for each user in the Chrome Application.
2. On server we will use this https://developers.google.com/identity/sign-in/web/backend-auth#calling-the-tokeninfo-endpoint to verify that token is valid. Because this is external call - we will cache valid tokens for an hour or so.
3. We will use https://github.com/brenden/node-webshot for generating screenshots.
4. We can use http://redis.io for storing validated google auth tokens (we have exp field for it). Plus we can use it also for storing screenshots as well. Say we will expire them once in day/week.

To make a load on the server a little bit less - I guess we should remove url parameters.

[artemgrygor]

@outcoldman looks good for me. I don't know 1 and 2 so will have a look on your prototype.
I'm ok with redis, want to work with it on nodejs.
Also we can store favicon too. There are no restrictions from the google, as far as I know. But why not, it will increase performance too. And will expire them once in month probably.

[outcoldman]

@artemgrygor I have setup milestone v1 https://github.com/deweyapp/dewey-server/issues?q=is%3Aopen+is%3Aissue+milestone%3Av1 , let's create new branch dev, keep all work in separate branches and for all work create PR.

[outcoldman]

btw, if anybody wants to help with all of that - feel free to do that

[outcoldman]

I have deployed version 4.0.0 which is basically one big change - using our own screenshot generating server.

If you want to help:

• As usual we will appreciate if you will contribute to code.
• Please consider donation https://github.com/deweyapp/dewey#donate - that will help us to switch to more powerful server.
• If you like - please share http://deweyapp.io to your friends and followers.