Discuss and encourage adoption of browserAction.openPopup()
#15
Comments
|
I have also run up against limitations of openPopup for the implementation of pie menus. https://news.ycombinator.com/item?id=27276093 DonHopkins | on: Use native context menus on Mac OS: I have always wanted to implement pie menus as a Firefox or Chrome extension, but I've never been able to find a sufficient API that enables me to pop up a modal dialog window at any place on the screen (specifically: centered on the current mouse cursor position, and in an operating system window that exists outside of the browser tab, overlapping all other windows). Or reshape it to any shape (or simply respect the alpha channel of the document and not show any chrome), so I can make a round or arbitrarily shaped window shrink wrapped to the items. https://developer.chrome.com/docs/extensions/reference/windows/#type-CreateType Is there a way with that (or any other) API to pop up and precisely measure and position an arbitrarily shaped (alpha channeled) window, without any other chrome or window frames? And then globally capture and track mouse and keyboard events? Does anyone know if there's now a way for a browser extension to do that in any browser? Or would it require hacking platform specific C++ operating system code? Here's a demo of an ancient implementation of pie menus I made for ActiveX around 1997, that shows pie menus with arbitrarily shaped windows: ActiveX Pie Menus: Demo of the free ActiveX Pie Menu Control, developed and demonstrated by Don Hopkins. https://www.youtube.com/watch?v=nnC8x9x3Xag ActiveX Pie Menus doc, examples, sources, etc: https://www.donhopkins.com/home/catalog/piemenus/ActiveXPieMenus.html https://www.donhopkins.com/home/catalog/piemenus/PieMenuDescription.html I did all the drawing with Win32 calls, so you could configure the fonts and colors and sizes and window shapes and styles, but you couldn't style everything arbitrarily with css, embed arbitrary web content, or anything nice like that. At the end of the demo video, I concluded that:
Of course it makes a lot more sense to draw and style the pie menus with the browser's renderer, but I still want the best of both worlds, where I can pop browser-drawn pie menus in arbitrarily shaped and positioned operating system windows, and track the mouse globally (capturing the mouse and keyboard events and receiving mouse motion and up and key events outside the window, to pop up and track sub-menus properly). JavaScript Pie Menus: Pie menus for JavaScript on Internet Explorer version 5, configured in XML, rendered with dynamic HTML, by Don Hopkins: https://www.youtube.com/watch?v=R5k4gJK-aWw aasasd: You'd probably want to use ‘native messaging’ from the extension for this: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Native_messaging I think it's sufficient to override the ‘contextmenu’ event and signal to your app where to show the menu and what to show in it. You can even implement the menu itself in something like Hammerspoon on Mac, with Lua APIs—dunno about Linux. However you'll have to define the whole menu yourself, since you don't have info on the browser's one. Though it might possibly be easier and more performant to have websocket communication with the app—it means the app will have to stay open, but that's not much of a problem in this case. OTOH I'm not sure that browsers won't start blocking requests to localhost servers, after the privacy issues. DonHopkins: That's in interesting approach. But my desire is to implement the rendering of the pie menus in the same browser that is popping up the pie menus. If it has to run in a separate app, then that app must embed its own separate browser than the one popping up the pie menus. Since it's a lot easier on the Mac to embed a Safari browser than a Chrome browser, it might be amusing to have Safari handling the pop-up pie menus of Chrome, but it wouldn't be the optimal solution of Chrome (or Safari or Firefox) drawing its own pie menus. Chrome's browser window extension api almost but not quite works for my purposes, and I wonder if or why it was a design decision not to make it more flexible. Security? Browser extensions, like clowns, can already get away with murder. aasasd:
Eh, I'm pretty sure it's quite a bit easier and faster to draw a radial menu with proper drawing primitives, like in Hammerspoon's API. Gotta admit I have no idea why you'd want animated pics or videos in such a menu. Though Hammerspoon's docs say that the drawing API does ‘support’ GIFs in some way, dunno exactly how. As a bonus, you could then invoke those menus in apps other than the browser, though you'd need some access to the UI, like AppleScript or accessibility APIs—which are typically neglected in cross-platform UIs. DonHopkins: How many developers and designers know how to draw using Hammerspoon APIs? Now how many know how to draw using web browser APIs? And how many people are actively maintaining and evolving Hammerspoon APIs, versus how many people are actively maintaining and evolving web browser APIs? It was quite clear to me in 1997 that I did not want to play catch-up to the web browser by programming an ActiveX OLE Control pie menu using the Win32 API and GDI to draw as nicely and as flexibly as web browsers could draw using HTML, CSS, VML/SVG/Canvas, etc. It's easier and faster to not reimplement (and maintain) the wheel. And who are you or I to say that the people designing pie menus don't want to use animated gifs, mpeg videos, Canvas, WegGL, or provide rich application specific real time visual feedback and animation during tracking, or do anything else in their menus that html is capable of? If you can't come up with a use case for menus with animated gifs, it's just a failure of imagination, not proof that menus with animated gifs are a bad idea. A web app should be able to easily use its very own web content in the exact same format to illustrate its menu items. In fact, I actually wanted to use animated gifs as pie menu items myself, and the demo video I linked to of my first Dynamic HTML implementation of pie menus shows animated gifs and html tables in the "Punkemon" pie menus, for example (at 3:00). https://www.youtube.com/watch?v=R5k4gJK-aWw&ab_channel=DonHopkins&t=3m (Sorry the graphics design and layout are atrocious, but I implemented that two decades ago, and the whole point was to piggyback on the next 20 of web development, not try to reimplement it all by hand.) Notice how (at 6:50) I use an XSL spreadsheet to transform the XML database of Punkemon characters into a set of illustrated pie menus rendered with html, tables, and animated gifs. In 2001, the web browser already has an XSL stylesheet processing engine built in, which you can use for dynamically rendering richly formatted pie menus from XML data, and it fully supports animated gifs, tables, and all html and css features out of the box. Does Hammerspoon? That's exactly what I mean about being able easily to use your native web content and technologies in menus. JavaScript pie menus (implemented circa 2001 for Internet Explorer with Dynamic HTML Behavior Components): https://donhopkins.com/home/PieMenu/piemenu.htc https://donhopkins.com/home/PieMenu/punkemon.xml https://donhopkins.com/home/PieMenu/punkemon.xsl https://donhopkins.com/home/PieMenu/JavaScriptPieMenus.html
|
|
It is my hope that we can figure out how to do what I was able to do two decades ago with Internet Explorer and ActiveX (and 33 years ago with NeWS and PostScript and HyperTIES), with modern web browsers today, but even better. More recently in 2013, I used the Safari WebView embedded in my own Mac application (based on the Slate window manager) to pop up arbitrarily shaped (and even partially transparent) pie menus overlaying all the other widows on the desktop, and globally track mouse input. The original Slate window manager was creating a WebView and hiding it, just to use its JavaScript interpreter for scripting the window manager. But I modified it to show the full screen browser window above all other windows, and respect the transparency of the page. I simply configured the "NSWindow *browserWindow;" that contained the "WebView *webView;" like this: browserWindow.opaque = NO; That caused WebView to respect the transparency of the web page it was showing, and use a transparent background color instead of drawing its own solid while background, and for its operating system window to float above all the other windows on the desktop. That is exactly what I'd like to be able to do with a modern web browser extension, but without having to run the web browser component in its own application, just so it has permission to configure the background transparency and perform global mouse tracking. Here is some more discussion on Hacker News about that: https://news.ycombinator.com/item?id=5861229 SimHacker on June 11, 2013 | on: Facts about Wayland vs X
https://news.ycombinator.com/item?id=18797587
https://web.archive.org/web/20180826132551/http://donhopkins.com/mediawiki/index.php/AQuery
https://news.ycombinator.com/item?id=11520967
https://news.ycombinator.com/item?id=13817649
[...lots more text and links...] |
|
To answer the question: "Why and how would you want to use pie menus in a web browser?": At the University of Maryland Human Computer Lab in the late 1980's, we developed a hypermedia browser called "HyperTIES" on the NeWS window system with UniPress Emacs, which supported pie menus for navigation and browsing, as well as user defined pie menus, running in embedded NeWS "applets" programmed and configured with PostScript code and data. (This predated Java applets, but both NeWS and Java were both written by James Gosling, as well as UniPress Emacs.) Designing to Facilitate Browsing: A Look Back at the Hyperties Workstation Browser. Abstract: Since browsing hypertext can present a formidable cognitive challenge, user interface design plays a major role in determining acceptability. In the Unix workstation version of Hyperties, a research-oriented prototype, we focussed on design features that facilitate browsing. We first give a general overview of Hyperties and its markup language. Customizable documents can be generated by the conditional text feature that enables dynamic and selective display of text and graphics. In addition we present: an innovative solution to link identification: pop-out graphical buttons of arbitrary shape. application of pie menus to permit low cognitive load actions that reduce the distraction of common actions, such as page turning or window selection. multiple window selection strategies that reduce clutter and housekeeping effort. We preferred piles-of-tiles, in which standard-sized windows were arranged in a consistent pattern on the display and actions could be done rapidly, allowing users to concentrate on the contents. [...] Pie menus to permit low cognitive load actions: To avoid distraction of common operations such as page turning or window selection, pie menus were used to provide gestural input. This rapid technique avoids the annoyance of moving the mouse or the cursor to stationary menu items at the top or bottom of the screen. [...]
Figure 1: Two articles are displayed. On the right side of the screen the title of the article is “Participating Organizations” which appears at the top left of the article, the page number (Page x of y) in the top right corner. On the left is displayed another article whose title is “Hubble Space Telescope — Main View”. The bottom of the screen is used for the standard commands e.g.: NEXT PAGE, BACK PAGE, INDEX, QUIT, etc…). When a user clicks once on the highlighted string “Faint object camera”, the corresponding definition appears on the four line area reserved in the low part of the screen. A double click brings up the Full Article on that topic, replacing the current article. [...]
The Hyperties browser uses pie menus as accelerators, to make commonly used commands quickly and easily available. A pie menu is a type of pop up menu whose selections are laid out in a circle around the menu center (18). The menu pops up centered on the cursor, so that each selection is adjacent to the cursor but in a different direction (Figure 1 and 6). A selection is made by moving the cursor in the direction of the desired selection, and clicking. Experienced pie menu users can make selections from familiar menus quickly and reliably without even having to look at the menu, because the menu selection depends on the direction between the two mouse clicks that invoke and select from the menu. The distance of cursor motion does not effect the selection, but the further away from the center the cursor is, the more precise the control of the selection is. The browser has a control panel at the bottom of the screen, with buttons showing the names of available commands, to turn the page, return to the previous article, show the index, etc. When users are browsing a document by pointing and clicking on highlighted text links in the main contents window, they move the cursor down to the bottom of the screen to press buttons in the control panel, and back up to continue browsing. The permanent display of those controls is important for the novice and occasional users. On the other hand, pop up menus reduce the distraction of moving the cursor by making these commands available wherever the cursor currently is. This reduces perceptual and motor load. Pie menus are arranged with their items in easy to remember directions. For example the BACK page turning commands are to the left (and the NEXT page is to the right) (Figure 6). This arrangement facilitates gestural input and encourages development of muscle memory. Experienced users can make gestural selections from these menus so comfortably and rapidly that it is often unnecessary to display the menu. This is called “mouse ahead display suppression”, and its point is to reduce the perceptual distraction.
Figure 6: This pie menu appears with a right button mouse click. It allows to turning pages, returning to the previous article, or jumping to the index. For example to turn to the next page, users drag the mouse to the left and release the button, thereby mimicking a page turning gesture. HCIL Demo - HyperTIES Browsing: https://www.youtube.com/watch?v=fZi4gUjaGAM&ab_channel=DonHopkins HCIL Demo - HyperTIES Authoring with UniPress Emacs on NeWS: https://www.youtube.com/watch?v=hhmU2B79EDU&ab_channel=DonHopkins Don Hopkins and pie menus in ~ Spring 1989 on a Sun Workstation, running the NEWS operating system. https://www.youtube.com/watch?v=8Fne3j7cWzg&ab_channel=CatherinePlaisant User Interface Strategies (UIS) 90 - Ben Shneiderman - Applications sections and demos. (HyperTIES demo starts at 12:40) https://www.youtube.com/watch?v=1uyO-xUTt6Y&ab_channel=CatherinePlaisant&t=12m40s |
|
I will be happy to open separate issues for each of the points I've raised, which relate to browserAction.openPopup and other APIs, and possibly a separate issue for extensions to browserAction.openPopup. Let's discuss how to organize these issues in today's conference call, please. I've signed up in the queue to discuss built-in and extension support for pie menus, tabbed windows, global mouse tracking, cursor position setting, customization, automation, permissions and security, and other issues relating to non-rectangular, partially transparent, full screen tracking user interfaces. In general, many of these are useful features provided by traditional window systems and operating systems, but have not yet been exposed to web browser extensions in a safe and standard way. Each feature is useful by itself, but they should dovetail together neatly. |
|
Interesting use case @SimHacker! My initial feeling is that there are two separate discussion points here:
I hope we can keep them separate to some extent - for the former, the API is already defined and I think we're pretty close to checking this one off. I think it would be a shame if we get too excited about what could come next and end up distracting ourselves from what should be an easy win. It doesn't work for all use cases (pie menus) but there are definitely cases where making browserAction.openPopup exposed to all extensions would make a huge difference. The second is no less exciting - it's something else we've wanted at 1Password. We display UI below fields to offer different filling options, and currently this has to be added to the HTML of the main page. This has a few drawbacks. For example, we have to work around #12 and display our UI in the top frame to avoid it being cut off by iframes of limited size. |
|
Yes, that's a good way to factor it into different issues relating to browserAction.openPopup. Not being able to control the position of the popup was the worst deal-breaker problem I ran into when trying to figure out how to implement pie menus as a browser extension. If that were possible, it would be easy to implement pop-up rectangular pie menus at least. Implementing arbitrarily shaped and translucent windows defined by the transparency of the web page is certainly possible and useful for many kinds of user interfaces beyond just pie menus. It would require its own permission of course, and the security implications need to be carefully thought out. I can also create separate issues relating to other APIs or usage cases, which would benefit from arbitrarily positioned and shaped windows, including: Menu management. So one shared implementation of menus can be used throughout the browser and by any web page or chrome. Capable of supporting both linear and pie menus, as well as hybrid combinations. (See the hybrid pie/linear menus in Blender, and the ActiveX, Dynamic HTML, and jQuery pie menus I've implemented.) Tabbed window management. Make tab and window management extensions more flexible, powerful, graphically pleasing. Supporting alternative approaches to managing windows and frames and tabs and embedding. Global input tracking. Capturing keyboard, mouse and other input. Relative mouse input mode. Cursor graphic and position feedback. Moving and constraining the cursor position. (These may relate to and extend some existing APIs, but I don't know which ones yet.) To illustrate what a menu manager could do, here's a great demo of the Blender Pie Menu Editor (which also lets you build and combine linear menus and other user interface widgets, too): https://www.youtube.com/watch?v=cQWwbBFQPrY&ab_channel=StevenScott https://www.youtube.com/watch?v=AIa-Yu4Z35U&ab_channel=blanchsb |
xeenon
changed the title
browserAction.openPopup()
|
@oliverdunk it appears that |
|
In Chrome it's only exposed in dev channel for normal (non-chromium) extensions: |
|
What is the expected dates for stable channel? Proposal document states 96 as target version? |
|
I am going to close this issue in favor of #160. #160 contains the discussion on how For the status of Chrome, I recommend to participate on the relevant issue on Chromium's issue tracker. |
One important API that is not currently available in all browsers is browserAction.openPopup, which provides a way to programatically open the popup associated with an extension. It would be great if through discussion here we could encourage adoption and also standardise some behaviour that works for as many use cases as possible.
We've wanted this many times over the years at 1Password and continue to encounter situations where it would be useful. Examples include:
There is clearly an understanding that this a useful API, even among browser vendors. In Chrome, it has previously been made available for the Google Cast extension to show a device list.
Firefox has implemented this API, but unfortunately the way it has been implemented does not work for us. As mentioned on MDN, "you can only call this function from inside the handler for a user action.". The thinking here makes sense, but unfortunately, user actions like clicking a button get lost if a message is sent to the background page. Ideally the browser would preserve the action across boundaries or allow us to call this API directly from a content script.
Tracking issues:
The text was updated successfully, but these errors were encountered: