MediaStream Image Capture API PTZ (Pan/Tilt/Zoom feature)

[riju]

Góðan dag TAG！

I'm requesting a TAG review of:

• Name: MediaStream Image Capture API (Pan/Tilt feature)
• Specification URL: Add pan and tilt constraints w3c/mediacapture-image#182
• Explainer, Requirements Doc, or Example code: https://github.com/w3c/mediacapture-image/blob/master/ptz-explainer.md
• Example Code: https://github.com/riju/WebCamera/blob/master/samples/panTilt/index.html

Code Snippet:

if (!('pan' in capabilities)) {
  return Promise.reject('pan is not supported by ' + track.label);
}
inputPanRange.oninput = function(event) {
  track.applyConstraints({
    advanced: [{
      pan: event.target.value
    }]
  });
}

Explainer/Motivation:
Many cameras have the ability to Pan and Tilt which is specially useful in the video conferencing(WebRTC), like steering the camera to face the speaker, etc.
FWIW Chrome has been using a private extension(webcam_private.idl) to satisfy this use case.

• Tests: https://github.com/web-platform-tests/wpt/tree/master/mediacapture-image
  Pan/Tilt Tests: [ImageCapture] Add pan/tilt constraint and wire in Linux/CrOS. web-platform-tests/wpt#15741
• Primary contacts: @riju, @YellowDoge

Further details (optional):

• Relevant time constraints or deadlines:
  @slightlyoff requested that we go through TAG review on this Intent to Implement and Ship thread
• I have read and filled out the Self-Review Questionnare on Security and Privacy. The assessment is here.
• I have reviewed the TAG's API Design Principles

You should also know that...

Image Capture API is already shipping in Chrome 59 on Android and desktop. Pan/Tilt feature is a small addition to the property set for capabilities, constraints and settings.

We'd prefer the TAG provide feedback as (please select one):

• open issues in our Github repo for each point of feedback
• open a single issue in our Github repo for the entire review
• leave review feedback as a comment in this issue and @-notify [github usernames]

[kenchris]

It is pretty hard to dig out from the example and a long Github discussion.

A small explainer showing the new API and explaining the use-cases would be much more welcome. We are all busy people and having to manually dig through a discussion thread is a bit too much. An explainer could summarize all this. You could also write a summary in the issue and link to that instead.

[kenchris]

FWIW Chrome has been using a private extension(webcam_private.idl) to satisfy this use case.

• Does this API differ from the private and presumable battle tested extension?
• pan and tilt values goes from what? 1-100? 0-1?
• Can all cameras tilt and pan as much? or is there a way to query their limits?
• It feels a bit weird that a pan value is considered a constrain and not just a control value.
• what was the conclusion of the "- Pan, Roll, Tilt vs Pan and Tilt" discussion

[riju]

Thanks @kenchris for the feedback. I have added code snippet and a brief explainer/motivation in the issue. Summary of the long Github discussion was mainly -

• Pan, Roll, Tilt vs Pan and Tilt
• Units to use, specifically - Arc seconds vs degrees.

[kenchris]

Units to use, specifically - Arc seconds vs degrees.

So which one are you using?

Also can you give a short summary on the pan/roll/tilt vs no roll

[riju]

FWIW Chrome has been using a private extension(webcam_private.idl) to satisfy this use case.

• Does this API differ from the private and presumable battle tested extension?

Presumably battle tested API gave Pan Tilt Zoom (PTZ). Zoom is already available from the start of MediaStream-ImageCapture API. Pan and Tilt are the recent additions and hence this issue

• pan and tilt values goes from what? 1-100? 0-1?

depends on camera

• Can all cameras tilt and pan as much? or is there a way to query their limits?

You can query the range. Different camera have different ranges. For example electronic Pan/Tilt do not pan more than 10 degrees.

• It feels a bit weird that a pan value is considered a constrain and not just a control value.
• what was the conclusion of the "- Pan, Roll, Tilt vs Pan and Tilt" discussion

Roll isn't commonly available in consumer webcams. Even though Pan/Tilt may be niche, we felt that Roll is even more niche.

[kenchris]

depends on camera

Actually reading the PR I read this:
1/3600th of a degree. Values are in the range from –180x3600 arc seconds to +180x3600 arc seconds

[riju]

Units to use, specifically - Arc seconds vs degrees.

So which one are you using?

arc-seconds.
I have summarized this in this comment

Also can you give a short summary on the pan/roll/tilt vs no roll

[yell0wd0g]

Image Capture provides a way to query the supported the MediaTrackCapabilities, by querying that on a video track you'll get for both pan and tilt their allowed ranges and steps, e.g for pan (same for tilt):

const trackCapabilities = imageCapturer.track.getCapabilities();
if (trackCapabilities.pan === undefined) {
  console.error('pan not supported, boo!');
} else {
  const maxPan = trackCapabilities.pan.max;
  const minPan = trackCapabilities.pan.min;
  const stepPan = trackCapabilities.pan.step;
}

All those numerical values will be in arc-seconds.

[dbaron]

So we had a brief discussion of the potential privacy issues in today's TAG meeting -- if hardware capabilities were to change substantially (e.g., laptop cameras that rotate a lot more than today), it feels like an implementation might want to add a separate permission prompt for the user to grant access to pan/tilt. So we'd like to make sure that the specification is designed in a way that would allow that to happen later on. It sounds (from a very brief look/discussion today) like the API is sufficiently asynchronous that that's the case, but it's worth thinking about a little more carefully.

[riju]

@dbaron / TAG : Just for clarification, does this mean a permission prompt first for getUserMedia() to access camera, and then another permission prompt to access the pan/tilt feature? Suppose a developer is making a PWA camera where she wants to use both pan/tilt feature and other MediaStream properties, does she ask user for 2 separate permission prompts?

[dbaron]

I think it's up to the user-agent / implementation to decide how to structure the prompts -- the key part is that the API should be designed in a way that allows for appropriate choices. This generally means it needs to involve promises rather than being synchronous so that a user agent might resolve the promise after prompting the user. It may also mean that the intent to use both the camera and pan/tilt should be present at the same time in case the user-agent wants to combine them into one prompt.

Of course, these demands may have tradeoffs with other desired characteristics of the API, so there might be reasons not to satisfy them. But they should be considered.

[plinss]

I presume that pan/tilt can be applied to any video track regardless of whether or not the imagecapture feature is being used? If so, it seems odd that it's defined here (along with most (all?) of the other capabilities/constraints defined)

[riju]

These new constrains apply to the live video feed, and that's covered in the Spec by making these an extension to the MediaStreamTrack: they are MediaTrackSettings, whereas others are only seen upon takePhoto() and they are PhotoSettings.

[kenchris]

Btw, the permission thing @dbaron mentions also applies to "zoom"

[cynthia]

@riju Sorry this took so long. It's the first time we mechanically (or logically, depending on the hardware implementation) allowed to "move" things in the physical world from the web, and firsts are always a bit scary.

We've discussed this in quite a bit of detail, and the group opinion (after a bit of back and forth) is that capture and control should be modeled in a way that it can be two distinct permissions; ideally requestable in a single call. Cases are where you would be fine showing one fixed (tidy) part of a room that is conference safe, and not allow access to the other side; this in native is covered by the conference software, but in the web we can't assume the software is trustworthy and will respect the user's preference.

The plumbing for implementations to be able to provide a way for the user to opt-out of this feature, while giving permission to the video stream seems like a valid use case, and we'd like to see this covered.

[riju]

Thanks TAG @kenchris, @cynthia, @dbaron for the feedback.

[kenchris]

Friendly ping @riju

[kenchris]

The TAG has decided that if we don't hear back before our next meeting we will close this with [resolution: unsatisfied] to get it off our radar and concentrate on more active tasks.

[beaufortfrancois]

@kenchris I will be drafting the owed explainer in the following days.

[beaufortfrancois]

May I ask you kindly to keep this issue opened as I'm collecting info? The current COVID-19 situation doesn't help. Thank you for your understanding.

[kenchris]

Sure, sounds good - good to know you are working on this. @torgo let's postpone looking at this for a couple of weeks

[beaufortfrancois]

The PTZ explainer is now available at https://github.com/w3c/mediacapture-image/blob/master/ptz-explainer.md

[torgo]

Hi folks - we had a good discussion on today's TAG breakout with @kenchris and I just had a few followup questions.

1. it was good to hear that there are mitigations against privacy issues being discussed, in particular a distinct permission prompt for pan/tilt/zoom and the idea that the feature will be disabled when the tab is not in focus.

2. the explainer includes some of this info but not everything - can you please make it more clear?

3. the explainer lacks a privacy & security considerations section (and lacks the word "privacy") and really needs to have this info explicitly called out for such a powerful API

4. what aspects of any of the above are intended for the spec as opposed to the implementation?

[torgo]

Hi @riju - we are just following up on this one on our TAG breakout call today. Has there been any update / progress on the questions listed above?

[riju]

Hi @torgo, not much. We just landed the platform support for Windows and CrOS. Linux was working fine. We are still discussing with the Privacy folks about the details and then we can update this audience.

[torgo]

Hi @riju – As far as the API design design goes, we are OK. However, we're going to mark this as "unsatisfied" because it really doesn't look like the security & privacy issues we've raised are being taken seriously. I would really encourage you to re-read our security & privacy self check and to add some explicit info to the explainer and to the spec covering abuse scenarios and mitigations against these scenarios.

[riju]

Thanks @torgo for the thumps up on the API design. We will take another look at the explainer soon.

[beaufortfrancois]

@torgo We've finally added Security and Fingerprinting sections to the explainer.

[riju]

@torgo : We have had some long discussions regarding this API in the WebRTC group and looks like now there's overall consensus among the stakeholders. Hopefully TAG is now satisfied with Privacy and Security information we have added in the PTZ explainer .