Upstream Shadow DOM spec to DOM/HTML Standard

[hayatoito]

Eventually, we should upstream Shadow DOM spec into DOM/HTML Standard.

Let's use this issue to discuss how we should proceed. That might not happen soon, but I hope we can start in the near future.

[domenic]

I took a quick look.

• SD section 2 could probably be moved into a subsection of DOM section 2
• We'd probably create a new top-level section in DOM for shadow DOM. Maybe after "nodes". Maybe at the end.
• SD sections 3 and 4 would move into the new top-level section.
• SD section 5 would need some work. It needs to be integrated into DOM section 3.
  • "In each algorithm in this section, the Window must be considered as if it were the parent node of the Document so that the Window also receives an event." I am pretty sure HTML has a similar sentence, but I cannot find it easily...
  • UI Events, Touch Events, and HTML will need patches for shadow DOM 5.5-5.9
• SD section 6 mostly gets upstreamed into HTML, some in Editing maybe.
• SD section 7 goes into HTML, creating several new element categories for 7.1, and changing the processing algorithms as in 7.2.
• SD section 8 goes into the new DOM top-level section on shadow DOMs
  • Some of the interface members on ShadowRoot need to go into other specs as partial interface definitions: activeElement and styleShets in HTML, element(s)FromPoint in CSSOM, innerHTML in DOM P&S, a few others... I think only host is left, actually.
  • I guess the definition of HTMLSlotElement and assignedSlot go in the HTML spec?
  • The blocklist of elements you cannot attachShadow too goes in HTML, or maybe becomes a DOM concept which HTML references.
• SD section 9 goes into a subsection of the new DOM top-level section on shadow DOMs

We should check with @annevk, but I would be OK with gradually moving over pieces, leaving pointers from the current spec into their new destinations. The trick is figuring out what are good chunks of work to do that in. For example, we could move over the HTMLSlotElement and assignedSlot stuff in one chunk and have it reference the shadow DOM spec until it's fully integrated. We could move over the conceptual stuff in another chunk. Etc.

[annevk]

I think a good place to start would be the "in a document" versus "in a x document" checks in the HTML standard. That is once the pieces that is actually biting implementers. I forgot what we decided on for x.

I think it's a good idea to do things in pieces.

[hayatoito]

As far as I know, for x, there are several candidates:

• in a document deeply: That's defined in Shadow DOM spec: http://w3c.github.io/webcomponents/spec/shadow/#dfn-in-a-document-deeply

However, in #362, I proposed connectedToDocument / disconnectedFromDocument.

The pros of in a document deeply is that it can be used as a replacement of both in a document and inserted into a document / removed from a document.

e.g.

• If the node is in a document deeply, ...
• When the node is inserted into a document deeply, ..

But if we use connected to a document, it might be confusing?

e.g.

• If the node is connected to a document, ...
• When the node gets connected into a document, ..

[annevk]

Isn't the problem with "deep" that if you have a closed shadow tree those nodes would be excluded, as is the case with deepPath? Would it not be "composed"?

[hayatoito]

So far, I do not feel any trouble by using the term of "deep" in the Shadow DOM spec.

When I have to consider a closed shadow tree, I am using "unclosed node" or "unclosed tree" in defining the behavior of a closed tree. That seems successful, so far. I do not have any trouble.

"composed" might be confusing because the Shadow DOM spec has a well defined concept of "composed tree". "in a document deeply" and "in a composed tree" are different concept.

But, I am fine with using composed here because it sounds more intuitive, I think.

Thus, the candidates would be:

1. in a composed document

   It might be confusing, due to the existing concept of composed tree. We might want to rename composed tree to something else, e.g. flattened tree, if we use "in a composed document" here.

2. in a document deeply.

3. connected to a document

[annevk]

So you are saying that deepPath uses the deep document, restricted to unclosed nodes?

I don't understand what the conceptual difference would be between "composed document" and "deep document" if deep document was not restricted to unclosed nodes. Which nodes would be in the former that are not in the latter? Or vice versa?

[hayatoito]

Sorry for confusing.

So you are saying that deepPath uses the deep document, restricted to unclosed nodes?

Basically, yes. (Though the spec does not use the term of "deep document")

• There is no concept of "deep document" nor "composed document" in the spec.
• The term of "in a composed document" is born in this thread and I meant we can use this new term as a replacement of "in a document deeply", if we want to avoid the term of "deep"
• There are concepts of "tree of trees", "in a document deeply" and "composed tree" in the Shadow DOM spec. Those are well defined terms.

I am thinking that we can replace existing terms with more understandable terms:

• "Tree of trees whose root tree is a document tree" => "composed document"
• "in a document deeply" => "in a composed document"
• "composed tree" => We might want to rename this to something else.

Note that "deepPath()" is only web-exposed API which uses the term of "deep". I am not sure how we should rename this or keep this as is.

[annevk]

My thinking with respect to the "deep" terminology was that it would be exactly like "composed", but restricted to "unclosed" nodes.

So most standards would operate on "composed documents", but APIs would be restricted to either "documents" or "deep documents", depending on the use case.

There was also the case of nodes that would not end up in the composed tree and how to address them. I don't think we had a solution for that.

[rniwa]

How about "in a connected tree scope" where "tree scope" refers to either a Document or a shadow root which is a node's ancestor, and "connected" means the tree scope is either a document or its host is "in a connected tree scope".

[hayatoito]

My thinking with respect to the "deep" terminology was that it would be exactly like "composed", but restricted to "unclosed" nodes.

I'm okay to use the deep terminology in this sense. But the current Shadow DOM spec does not follow this rule, e.g. "deep child": http://w3c.github.io/webcomponents/spec/shadow/#dfn-deep-child

If we reserve the "deep" terminology for this purpose, "deep child" or other similar terminologies should bee renamed to something, such as "composed child" ?

So most standards would operate on "composed documents", but APIs would be restricted to either "documents" or "deep documents", depending on the use case.

I am fine with this idea, basically. From my experience, most standards could operate on "a composed document". It's rare case that APIs would be restricted to "deep documents".

There was also the case of nodes that would not end up in the composed tree and how to address them. I don't think we had a solution for that.

"in a composed document, but not in a document composed tree" could work, I think.
http://w3c.github.io/webcomponents/spec/shadow/#dfn-document-composed-tree

It this is too long, we have to invent yet another terminology as an alias for this long condition.

[annevk]

@hayatoito yeah, "deep child" would have to become "composed child".

@rniwa is "connected" a replacement term for "composed" or "deep" (with "deep" meaning "composed" excluding "closed")?

[rniwa]

"connected" would mean it's in the document or your shadow root's host is in the document regardless of whether any of shadow root inside which you reside is closed or not. So I think it corresponds to "composed" in your terminology.

[hayatoito]

If we can agree that we use "in a composed document", let me do a batch internal renaming refactoring in the Shadow DOM spec as follows:

• a composed document: The new term, which is equivalent to "Tree of trees whose root tree is a document tree"
• in a composed document: This is exactly equivalent to "In a document deeply"
• deep XXX: => Will be renamed to composed XXX
• composed XXX: => Will be renamed to flat XXX

This internal renaming does not affect any web-facing APIs.

I prefer "in a composed document" because it is short, relatively. "Shortness" is good here because it is expected that the term would appear in many places.

Does it sound good?

[hayatoito]

Renaming was done at 9b7f16e

[annevk]

I want to start working on #57. My tentative plan is:

• Introduce terminology in the HTML standard and later upstream that to DOM. (Terminology as discussed per above, perhaps with adjustments as necessitated.)
• Provide one or more PRs for the cases we know how they should work and open issues against the HTML standard for cases we don't know (e.g., it seems reasonable to open an issue for currentScript although I think we probably want to resolve it similarly to activeElement).

This will shift some of the development work of Shadow DOM elsewhere, but I think that's okay, since it mostly affects features defined elsewhere. Not too different from the collaboration between Service workers, Fetch, and HTML, or HTML and JavaScript, etc.

I will copy @hayatoito for review. Anyone else who wants to tag along let me know. We could perhaps create a GitHub team called "shadow" that we tag on any such issues and PRs, if that's needed.

[hayatoito]

Sounds great. I would appreciate if you could work on this. I'm totally fine with shifting the development elsewhere.

[annevk]

@hayatoito so another thing here what would help is updating the Shadow DOM specification to build on concepts established elsewhere. E.g., now DOM has "get the parent", Shadow DOM could use this to define the event path. Other extensions to event dispatching could also be made a little bit more explicit and such. Is this something you plan on doing or are you hoping that will be taken care of during migration? It seems to me we could do it during migration, but there'll be a lot of changes then.

[hayatoito]

Sure. I am happy to update the Shadow DOM spec so that it will use an established new concept as much as possible even if we are in the migration period.

Please feel free to file an issue, e.g. "Use get the parent in a shadow spec", or ping me if you are introducing a new concept in your commits. I would like to get notified somehow.

[annevk]

I've started adding some shadow-related concepts in DOM by the way:

• whatwg/dom@b2ee400
• whatwg/dom@1699495

[annevk]

@domenic @hayatoito do you think the best approach for slots is that DOM defines a concept of a slot element and HTML says that <slot> is it? And we mention in DOM perhaps that <slot> is really the only thing out there. The alternative is directly referencing <slot> in HTML which might also be okay. DOM and HTML are intertwined anyway.

[annevk]

My plan is to figure out slots first, including how the HTML parser needs to change, etc., and only then take over event handling. Slots are important for events, so that seems like the best way to do this incrementally.

[hayatoito]

Sounds good to me. An event, especially get the parent algorithm for Node, is only the consumer of slots in DOM Standard, I guess.

[domenic]

That plan sounds good to me, kind of like how DOM takes care of some of the <template> stuff. But, if it becomes convoluted, referencing HTML directly is also OK.

[annevk]

I had added language for slots to the DOM Standard. Next is figuring out the superglobal slot attribute (how it fits into the HTML Standard), then define the HTML slot element, and then events. Layout likely needs to be done by someone else, but I'm happy to add all the hooks to DOM that requires, of course.

Am I overlooking anything?

[hayatoito]

I have done most of 3 too: https://dom.spec.whatwg.org/#finding-slots-and-slotables.

Looks good to me. A nice simplification.

[domenic]

At some point we should start removing stuff from http://w3c.github.io/webcomponents/spec/shadow/ and pointing those sections to DOM/HTML.

[hayatoito]

Yeah, I have already started to remove some concepts from the Shadow DOM spec and started to use the newly defined concepts in DOM Standard, though, the spec is not always up-to-date.

Let me clean up the spec periodically.

[chaals]

Domenic, Hayato, please hold off removing stuff from the specs. W3C has been working for some time to try and make its specs more modular instead of more monolithic, so the question is now in front of the Working Group.

[domenic]

Chaals, these specs cannot be more modular; they are deeply intrusive into the existing standards and exist largely as monkeypatches. It's important to get them into their proper contexts.

This has always been the goal of web components; template was the first success, and custom elements and shadow DOM are following its footsteps.

[annevk]

I've started making the necessary changes to HTML's algorithms to account for shadow trees by the way. Easiest way to stay informed is following https://github.com/whatwg/html as there are many changes necessary and landing them all in one go does not really seem feasible, at least not without lots of rebasing that I'm not particularly keen on (and it would make reviewing a pain too).

[hayatoito]

The status of upstreaming:

• 2. Composition (=> DOM Standard)
• 3. Distribution (=> DOM Standard)
• 4. Flattening (=> CSS Scoping. I have to re-check if it reflects correctly)
• 5. Events (WIP. Partially done)
• 6. User interaction (Not yet)
• 7. HTML Elements in Shadow Trees (=> HTML Standard, WIP)
• 8. Elements and DOM interfaces (Mostly done => DOM Standard and HTML Standard, DocumentOrShadowRoot is not yet)

I could not spend much time on maintaining Shadow DOM specification recently.
Some sections might be outdated if the sections are already upstreamed.

[hayatoito]

The status update:

• 2. Composition (=> DOM Standard)
• 3. Distribution (=> DOM Standard)
• 4. Flattening (=> CSS Scoping)
• 5. Events (=> DOM Standard, UI Events, Touch Events)
• 6. User interaction (Not yet)
• 7. HTML Elements in Shadow Trees (=> HTML Standard, WIP)
• 8. Elements and DOM interfaces (Mostly done => DOM Standard and HTML Standard, DocumentOrShadowRoot is not yet)

The remainings are Section 6, 7 and 8.

[hayatoito]

Let me clarify my upstreaming plan:

I am trying to upstream Shadow DOM into DOM/HTML to confirm that it can be integrated with DOM/HTML properly. The reason I could not maintain W3C Shadow DOM specification to reflect the latest change which is happening in DOM/HTML Standard is my bandwidth and its difficulty. It is difficult to pick up necessary parts nicely with a clear separation.

Let me try to spend my time on it, but I would also welcome contributions. If someone could work on downstreaming to W3C Shadow DOM specification from DOM/HTML Standard, it would be greatly appreciated.

I'm afraid that we have to copy almost all of "3. Events" and "4. Nodes" section, as is, from https://dom.spec.whatwg.org/, at least.

[annevk]

Why would we maintain a copy? That seems way confusing.

[slightlyoff]

Hey @annevk:

It might be confusing, but such is the price of peace! We (the Chrome team) feel deeply grateful to our colleagues at Microsoft (and other places) who have worked with all of us to keep Web Components on track. Part of the agreement that has allowed them to participate is that we maintain a version of this spec which can be run through the W3C process inside of WPWG. That means we'll carry the patch. More work, perhaps confusing, but necessary.

Thanks for understanding.

[annevk]

@hayatoito do you have a pointer for the Touch Events work as I don't find anything about shadow trees in https://w3c.github.io/touch-events/? Same for UI Events actually, nothing about shadow trees in https://w3c.github.io/uievents/. I'm also missing a pointer to w3c/DOM-Parsing#21 for innerHTML.

https://github.com/whatwg/html/labels/topic%3A%20shadow is another list of outstanding issues.

Should we perhaps create a new tracking issue for all these items so we have a clean slate for all upstream work?

[annevk]

It seems we also have #495 which attempts to track something similar perhaps? @rniwa thoughts on where to track all these issues?

[hayatoito]

Regarding TouchEvents, it looks what I have done is to make TouchEvents 'composed' events, and add event retargeting process.

• w3c/touch-events@eeed53b
• w3c/touch-events@5999f27

Regarding UIEvents, it looks what I have done is to make some UIEvents 'composed' events.

• w3c/uievents@7341238
• w3c/uievents@d075314

Hmm, I forgot to add retargeting process there. :(

Regarding w3c/DOM-Parsing#21, this was not fixed. I remember that we had a discussion about whether we should add innerHTML to DocumentFragment, rather than ShadowRoot, or not. However, I guess that was not solved due to the lack of follow-up.

Should we perhaps create a new tracking issue for all these items so we have a clean slate for all upstream work?

Yes, that would be great. Recently, I couldn't spend much time on upstreaming tasks. Please feel free to create a new tracking issue.

[annevk]

I remember that we had a discussion about whether we should add innerHTML to DocumentFragment, rather than ShadowRoot, or not.

We decided not to do this, since DocumentFragment doesn't always have a host like ShadowRoot does. This makes the "context node" question for the HTML parser much more complicated.

[annevk]

Closing in favor of #661.