Module Normalization

[dgeb]

Rendered

[ghost]

Looks very good in the general, just a small thing:
Is the module directory addition really necessary? Is re-use of app/addon possible?

How would routable components look with the proposed structure?

[rwjblue]

@martndemus - The modules directory replaces the app and addon concepts. Removing a fairly large annoyance and learning curve when developing addons...

[dgeb]

Thanks @martndemus

Is the module directory addition really necessary? Is re-use of app/addon possible?

To expand upon @rwjblue's comment:
Using the same directory for both apps and addons would mean that addon modules aren't closed over as they currently are (although they could still declare a private namespace internally) and special export-only modules would not be needed to make addons accessible to their parent app.

Also, a new directory would be required for a transition strategy (i.e. to allow Ember CLI to understand both the current and new module structures simultaneously).

How would routable components look with the proposed structure?

They would co-exist alongside routes in a nested namespace. It's TBD whether they would use the type of component or whether we would flag them with a new type to distinguish them. I find routable-component verbose - so perhaps an alternative?

[jschilli]

I'll reread this again but WRT addons mixing into the consuming app namespace:

Today a consuming app can override consumed addon types e.g. addon has templates/login.hbs and the consuming app declares it's own to 'override' the addon's template.

the addon is responsible for displaying said template.

How does that work in this model?

We rely heavily on this behavior to extend addons (which we use heavily as a pre-engine way of breaking up our large app)

[scottmessinger]

I really, really like the dot naming convention with namespace, name, and type. It's the first naming/module convention that has felt spot on.

The bucket idea where some folders contribute to the lookup path and others (prefixed with - do not) really threw me for a loop. I've never heard of a system where some folders don't contribute to module path -- either they all do or none of them do. The confusion caused by the newness of this approach could very well be worth it, but I think many, many people will be surprised that a seemingly insignificant - caused the module lookup to break. I certainly would be. One possibility is to make the bucket folders have an even more surprising naming prefix/postfix so the magic of it is glaringly noticeable and prompts new devs on an Ember project to ask a question (or go to the docs). For instance, +bucketName+ or __bucketName__, or _-_bucketName_-_. Another idea is to have folders prefixed with bucket not contribute to module's lookup path. e.g. (bucket.components).

[chadhietala]

Is this map going to be created at build time? I'm pretty convinced that none of this string parsing should happen at runtime. There is no reason you can't build the map at build time and hydrate the resolver with the precomputed map.

[rwjblue]

Yep, we absolutely plan to do this at build time so the runtime resolver has much much less work to do.

[chadhietala]

👍

[aaronbhansen]

We've locally implemented almost this exact pattern for the last two years and have really enjoyed using it. We didn't have the name.type pattern implemented in our custom resolver, but we also haven't felt a strong need for it as auth.js was enough of an identifier. It feels like the name.type pattern could help clarify some files purpose, but there are still components, such as new-post, that have files with type only names (component.js and template.hbs). I would rather have the components nested in a folder than flattened out to include the name and type in it, but it does feel there is a discrepancy for files in some buckets verses others.
Overall the buckets have helped really managed our projects files in a clean structure. Putting global types in folders in the root really ensures that the routes remain clean and only contain route specific concerns. Nesting components, helpers, etc in a route when they are specific to a route or parent route helps maintain concerns at those levels. Nice write up on the RFC. (Part of our original proposal which is pretty close to the RFC's bucket example)

[dgeb]

I've never heard of a system where some folders don't contribute to module path -- either they all do or none of them do.

@scottmessinger I've probably done a poor job explaining the concept of "normalized" modules because I actually agree that this is important.

ES modules will still mirror file locations on disk, and can be imported into other modules accordingly. Therefore, your imports would still include any bucket segments in module paths.

However, the resolver will be looking up modules based upon the normalized form of module paths. The resolver will reference a mapping maintained between normalized modules and file-based modules. This mapping will be created at build time to minimize work at run time.

[chadhietala]

Overall this looks amazing and addresses all of the issues that I had mentioned in #44.

[chadhietala]

Is there plans to EOL existing project layouts in 3.0.0 or is the plan to support 3 project layouts?

[dgeb]

EOL hasn't been specifically discussed, although I'm sure that existing project layouts will be supported for quite a while. It seems critical to be conservative here, especially regarding addons which may not be under the control of a project's owner.

[knownasilya]

How do normalized and file based paths affect buckets? The explanation above needs an example I think.

[workmanw]

👏 👏 There is a lot of great stuff in here and I think it strikes a good balance between a lot of tension points.

Huge 👍 to the "buckets" idea. We've avoided the current pod structure for two reasons, both of them feel solved by this RFC.

1. We've never liked the idea of having models sit at the same hierarchy as components. Things like models, adapters, serializers, and transforms have always felt like a "layer" in our application. Being able to isolate them outside of the route/namespace hierarchy [now modules] is very desirable.
2. Having hundreds of files called template.hbs or component.js has been a worry. Having meaningful filenames makes it a bit nicer when you're working in your editor, but more importantly looking at a stack trace in chrome which only shows the filename (not the filepath).

One question I do have is regarding styles (CSS, SASS, etc). What do you see is the impact for style based addons (eg ember-cli-sass and ember-cli-less)? Would they be able to easily take advantage of this structure? Perhaps the onus is on the individual addons themselves to adopt support of this. Ultimately I'd really love to have post-editor-button/style.sass which is automatically imported into the SASS tree.

[grapho]

I like the bucket idea.. but for routes that have child routes.. should those child routes not go within a new -routes directory? the pattern is consistent with -components and -helpers on each route level.. but not with the routes themselves?

I have a feeling that if the fractal pattern is solid and consistent throughought, then refactoring and moving files around would be much simpler?

[csantero]

How does this work with nested component modules? If the magic-ui addon has a component corresponding to the module my/awesome/select/component, what does the invocation look like in hbs?

[dgeb]

@csantero One of the design constraints seems to be a blocker:

No slashes in component names. The existing system allows this, but we don't want to support invocation of nested components in Glimmer.

I wasn't present when this constraint was formulated, so I don't know the full rationale. However, it seems technically possible to support nested namespaces à la {{magic-ui::my:awesome::select}}.

[runspired]

this is the biggest blocker on this syntax for me. Across all projects I've been involved with this is an unmanageable breaking change with no path forward without support for better component namespacing.

[knownasilya]

Agreed. Didn't realize that this was one of the limitations.

[rwjblue]

@runspired - This is absolutely not a breaking change. All existing structures will continue to work.

[runspired]

@rwjblue I left a longer note below. This may not be a breaking change for curlies, but as things stand curlies would be relying on (assumed to be deprecated) lookup patterns, and there is no path forward from curlies to angle bracket components that use this pattern.

[rwjblue]

To my knowledge there are no plans to deprecate curly components. Curly components will have capabilities that are unavailable in angle bracket components (i.e. tagless/fragment)...

[runspired]

while I get there are a few behaviors we will need to leave curly components behind for, seems odd to box some forms of components into curlies just to make them more composable..

[rwjblue]

My general experience sides with @dfreeman, the vast majority of my current nested components would be solved very nicely by the current proposal.

[csantero]

Our company has built up a large inventory of utility/infrastructure components that live in a couple monolithic shared addons used by our apps. There are so many of these components that we feel that nesting them is a big win for usability. If nested namespace angle-bracket components aren't possible, we will have to either suffer the headaches of having dozens of components at the top level, or just forgo the potential benefits of angle brackets.

[mixonic]

@grapho as much time as I've spent on this, keeping all the details in one's head is quite tricky. I confirmed my understanding with @dgeb and let me see if I can answer your query:

I like the bucket idea.. but for routes that have child routes.. should those child routes not go within a new -routes directory? the pattern is consistent with -components and -helpers on each route level.. but not with the routes themselves?

From the RFC:

It's proposed that we introduce special "bucket" directories for organizing files. These directories should be prefixed with a - to indicate that they will not be used to inform module names, namespaces, or types. Every module defined within a bucket simply flows through to the bucket's parent namespace.

Emphasis mine. Buckets are an ergonomic addition, not a semantic one. At the core of this proposal, they have no meaning. For example take this route:

modules/cars/-components/make-and-model/component.js

Note that the word component shows up twice. Each module has three parts, and this one would break down like so:

modules/cars/-components/make-and-model/component.js
        ^namespace       ^name          ^type

But this works with or without -components. Thus for your query about routes, note that the RFC always has the type in the file name:

modules/booksellers/by-city/route.js
        ^namespace  ^name   ^type
modules/booksellers/regions/by-city/route.js
        ^namespace          ^name   ^type
modules/booksellers/regions/by-city/template.js
        ^namespace          ^name   ^type
modules/booksellers/regions/by-city/-components/city-map.template.hbs
        ^namespace                              ^name    ^type

The above would describe a route booksellers.regions.by-city with a local-lookup {{city-map}}. Local lookup is performed by taking the namespace + name of the route's template and using it as the namespace for the component's name. let localLookupNamespace = invokingTemplateNamespace + invokingTemplateName;

And note that the type of the template is template, not component even though it is in the -components/ directory.

Later in the Buckets section @dgeb says:

Ember CLI should probably enforce the use of these buckets as well as the types of modules allowed in each bucket.

So in Ember-CLI, we would lint/assert that you only put modules of the type template and component inside the -components/ directory. But this is a build-time assertion, not a part of the runtime system. At runtime, the -buckets are not used.

We do not need to endlessly nest -route/ directories because nested directories only build the namespace and name for the module. The type will always be defined in the file name (route.js or some-name.route.js).

[mattmcmanus]

Overall, I like this approach. I love the name ${namespace}/${name}.${type}file naming pattern

I share @scottmessinger's reaction about the buckets and the -. The dash feels kind of like a smell to me, but I can't come to any conclusion about an alternative that provides equal benefits. Maybe just calling it out more would be all that is needed. _Components, _Helpers?

I love the idea of being to have unit.test.js and integration.test.js alongside of component.js. I think having acceptence.test.js in the same directory as a route.js would also be quite nice.

As for routable components, I don't think any particular distinction in folder structure is necessary. I've found myself preferring the react terminology of "container" components and think that would work well with the component names, i.e.,: new-post-container.

This is my first RFC comment. I hope it's helpful!

[runspired]

Overall I think this is a step forward over other "pods" RFCs, but I think this is incomplete in a critical area because of the desire to "clean up" the template invocation of angle-bracket components.

Here is an example of a common and useful structure that does not seem possible to convert nicely into this new structure.

routes (aka 'modules')
  /blog (namespace)
    /posts
      template.hbs (uses post-list)
      /index
         template.hbs (uses post-list)
      /single
         template.hbs (uses single-view)
      /edit
         template.hbs (uses single-view)
      /components
        post-list/
          component.js
          template.hbs
        single-view/
          component.js
          template.hbs

This component use pattern doesn't seem possible in the proposed namespace pattern without sub-namespacing, the implementation of which according to @dgeb's comment above isn't guaranteed.

e.g.

{{blog::posts::post-list}}

These components are problematic because they are all part of posts conceptually, not blog, not local and not global. The current proposal would likely box you into stuffing a ton of components onto the blog namespace, but even then, a flat structure becomes required.

doesn't work

modules/
  blog/
    -components/
       posts/
          post-list/
            component.js
            template.hbs
          single-view/
            component.js
            template.hbs

does work

modules/
  blog/
    -components/
        posts-post-list/
          component.js
          template.hbs
        posts-single-view/
          component.js
          template.hbs

This gets us right back to the same problem we had that led us to dream up pods: giant unmanageable component directories, albeit (slightly) improved by the top level namespace breakout and some components being able to be local.

Interop with engines and the ability to break out portions of an app would be a lot nicer if namespaces carried down the route hierarchy (e.g. were nestable) instead, but for deeply nested pods I get that that's a performance cost we want to avoid.

Although verbose (and more verbose maybe than needed), this syntax below is something I really liked about the current pods structure, because it makes the component's location explicit. You don't need to reason about whether a component is in a local, namespace, or global directory, and it let's components
that are in between be found just as easily.

{{#blog/posts/components/post-list}}

{{/blog/posts/components/post-list}}

I would like to decrease the verbose nature of the above, ideally it be nice to be able to do {{#blog/posts:post-list}}{{/blog/posts:post-list}}.

Two colons?

I'm not sure why two colons were preferred to one in the syntax, but at least at the moment
one seems just as feasible and a little more ergonomic to me over two.

Angle Bracket Components

How does this work for angle bracket components? Is it identical?

<posts::post-list></posts::post-list>

It seems to me that if the above is the syntax for angle brackets, then there isn't much reason why
the syntax couldn't be expanded to allow for paths in components, and address this organizational matter nicely.

e.g.

<blog/posts:post-list></blog/posts:post-list>

[runspired]

Another thought, I would prefer that all of a type go into a bucket.

example 1

modules/
  posts/
    -components/
       post-list.js
       single-post/
          component.js
          template.hbs

not

example 2

modules/
  posts/
    post-list.component.js
    single-post/
       component.js
       template.hbs

By enforcing buckets I think we not only make our lives easier programmatically, but we increase the developer ergonomics a great deal. In example-2, I don't know that single-post is a component and not a child route until I expanded the directory. Buckets bring sanity by letting the pod structure accurately and clearly reflect the nature of your nested UI.

[mattmarcum]

Wouldn't it be weird having two different naming conventions, name/type.ext and 'name.type.ext' files, in ember? I'm thinking about this w/r/t convention over configuration and bike-shedding here. Which pattern would ember-cli use for its default generators?

Doesn't the name/type.ext pattern work against design constraint 9: "Avoid the "titlebar problem", in which many files are all named "component.js" and you can't tell them apart in your editor." It might seem nit-picky but that's a big reason why I've avoided the pods structure so far. What benefits does name/type.ext bring - is it just personal preference for organization?

[dgeb]

Another thought, I would prefer that all of a type go into a bucket.

@runspired That could work if there is a 1:1 mapping between buckets and types allowed in each bucket. This option was explored in an alternative proposal that was dropped. This form could technically coexist with the normalized form if we instead dropped support for multi-type buckets like -data (which can contain models, adapters and serializers) and -components (which can contain components and templates). Many developers seem to prefer the ergonomics of the more flexible multi-type buckets.

[dgeb]

Wouldn't it be weird having two different naming conventions, name/type.ext and 'name.type.ext' files, in ember? I'm thinking about this w/r/t convention over configuration and bike-shedding here. Which pattern would ember-cli use for its default generators?

@mattmarcum this seems like an entirely valid concern, and is the reason I added this unresolved question to the RFC: "Should we tighten the flexibility of the alternative patterns? For example, we could require top-level buckets or emit warnings when certain naming patterns are used with certain types."

What benefits does name/type.ext bring - is it just personal preference for organization?

It's a more concise format that avoids repetition. And the "titlebar problem" isn't present in all editors, so it won't be a major issue for some.

With all that said, we may come down on the side of choosing a single convention rather than allowing all the flexibility presented in this RFC. Thanks for sharing your perspective.

[knownasilya]

I don't think we should be designing a folder structure around an issue with file editors. Those issues can be and should be addressed on the editor side. This is possible now, and I know Atom has some plugins that address this.

[runspired]

@dgeb I prefer multi-type buckets too. I think I worded what I meant poorly. Basically, what I'm suggesting is that only routes can folders within another route, all other "types" must be in a bucket of some form. I think This would make your organization follow your ui, and help keep the ui nesting / organization clear at the directory level.

[dgeb]

I share @scottmessinger's reaction about the buckets and the -. The dash feels kind of like a smell to me, but I can't come to any conclusion about an alternative that provides equal benefits. Maybe just calling it out more would be all that is needed. _Components, _Helpers?

@mattmcmanus I don't particularly like the bucket prefix either, but we do need some rule to prevent conflicts with namespaces. I'm open to alternatives.

One alternative I've considered is to allow buckets only at the top-level, with no prefix, so the the entire top-level set of directories "drops out" of the normalized form. This of course would also require usage of those top-level buckets, since every directory at that level would need to be considered a bucket, not a namespace.

As for routable components, I don't think any particular distinction in folder structure is necessary. I've found myself preferring the react terminology of "container" components and think that would work well with the component names, i.e.,: new-post-container.

This is my first RFC comment. I hope it's helpful!

Definitely - thanks for chiming in!

[luxzeitlos]

Just to ask, if - indicated a bucket, how can addons have a -private namespace?
Also I would prefer something that is not occupied, and using - could make it hard to clarify the difference between dasherized names and the bucket indicator. Maybe use @ or I personally like > or maybe an arrow -> because its not used in filenames for anything as far as I know and it does not blocks readability.

-components
>components
->components

-my-bucket
>my-bucket
->my-bucket

Overall I love this RFC.

[trabus]

I don't have much to add that hasn't already been said, but I love this RFC. Great solution, and it gives us a path forward with backward compatibility until Ember 3.0. (and likely after with an addon).

[chadhietala]

@rwjblue @dgeb @stefanpenner In general I'm 👍 with what this RFC lays out. I'm wondering if something more can be done with the mapping of normalized module name to path on disk. One of the challenging things about the Linker/Packager stuff is there is no "main entry" of ember apps to start dependency resolution from. If we generated a main file that imported an app/engine's files like the following we could solve this problem and also achieve the mapping.

// dummy/main.js
import dep0 from 'dummy/main.app.js';
import dep1 from 'dummy/main.router.js';
...

export default {
  'dummy@application:main': {
    path: 'modules/dummy/main.app',
    factory: dep0
  },
  'dummy@router:main': {
    path: 'modules/dummy/main.router',
    factory: dep1
  },
  ...
}

To prevent evaluating all of the objects up front we could create lazy accessors to the modules like @stefanpenner experimented in emberjs/ember.js#11576. Or we bring back the minimatch wrapInEval trick. The resolver can than just pull in this file for each namespace.

The only other tricky thing is allowing addons to diverge their package name from their namespace. While it's still doable the Packager/Linker stuff is going to need to do a song and dance to figure out the mapping of import select from 'power-select' to path.join(process.cwd(), someNamespaceAwareAddonMapper('power-select') to discover the file on disk. Traditionally we would just use the normal node module resolution, but it appears we are a little bit ❄️ in this regard.

[stefanpenner]

The only other tricky thing is allowing addons to diverge their package name from their namespace.

i think we should disallow this entirely

One of the challenging things about the Linker/Packager stuff is there is no "main entry" of ember apps to start dependency resolution from.

aren't we able to just treat the whole engine/app as a retained? Or am i mis-understanding.

[chadhietala]

@stefanpenner We can connect all files in an app/engine as root nodes in the graph. The advantage of having the main file is that it is static and thus there is no special logic that converts a namespace into a retained graph. The convention would be each app/engine has a main file which is it's module mapping and it's root nodes.

This is getting orthogonal but with a main you could not only tree shake but also do dead code elimination like rollup. Since module resolution goes through a resolver you technically can have 1 module that exports the mapping. The mapping just points to inlined modules in that 1 module.

[stefanpenner]

The convention would be each app/engine has a main file which is it's module mapping and it's root nodes.

don't we already need to treat everything in an engine as root nodes? Most files aren't connected via module imports, as they are loaded by DI.

[chadhietala]

Yes you have to root all of the files. I'm just saying you can cleanup the walk if you can start from a real main instead of a "virtual main" that requires a filter and connect pass. Ideally this is the simplest code to build the graphs.

const resolve = (id, imports) => {
  imports.forEach((import) => {
   graph.add(import, { namespace: ns(import) });
   graph.setEdge(id, import);
   resolve(import, modules[import].imports);
  });
}

entries.forEach((namespace) => {
  let mainPath = 'modules/' + entry + '/main';
  let main = modules[mainPath];
  graph.add(mainPath, { namespace });
  resolve(mainPath, main.imports);
})

When you go find reachable/unreachable/intersection of the graph you have to run something like postorder on each node that is part of the root. This means you must individually track the root nodes. With a main you simply use that as the entry node and run postorder on it as it import all the modules. Just trying to front load the complexity brought on by many roots due to DI.

[sglanzer-deprecated]

For addon namespaces, is it possible to use the same namespace for multiple (logically related) addons? Alternatively, could multiple "main" components be exported from an addon?

We have an addon that acts as a core component library and a number of extension addons that can be optionally included and I'd prefer that all of these either follow the {{lib-specific-comp}} or {{lib::specific-comp}}, not a mix of the two styles.

[stefanpenner]

some thoughts:

1. having multiple ways to define the same thing feels like we are attempting to satisfy too many masters, we should aim to avoid this, and choose a single approach. Additional flexibility could be added at a later time, but in the interim we can instead be conservative. Specifically: namespace/name/type vs namespace/name.type.js
2. namespace/name/type vs namespace/name.type ambiguity, type vs non-resolvable is currently ambiguous. namespace/name/transition is reasonable to have as a file, unrelated to ember but imported by another file (like a component or controller), but when liquid-fire is introduced that file now has meaning and could be resolved my LF by mistake. If instead, any file with name.type.js was reserved to be resolvable, no such ambiguity would be possible.
3. buckets may add organizational value, but currently feel premature, I believe introducing them later is at no risk so let me recommend, we merely reserve folders prefixed with - and re-evaluate buckets if needed at a later time.
4. modules/lib feels like it will instantly become a junk-drawer, I prefer co-locating my "libs" with contextually relevant code ( as described in 2.) rather in an independent hierarchy.

[dgeb]

Thanks to all for your feedback.

At the last face-to-face meeting, the core team nixed the concept of "buckets" as defined in this RFC. It was decided that the sleight of hand required to allow buckets to be used for organization only, and not for resolution, could create confusion. Modules could conflict across buckets, because they
could have matching namespaces, names, and types. This kind of conflict could not be allowed, so developers would need to understand too much about the resolution strategy to make it ergonomic.

I am working on a new proposal that builds on many of the concepts in this RFC, but replaces buckets with a new "collection" concept. Collections provide the same organization benefits as buckets but do not drop out of normalized namespaces or the resolution process. Instead, rules for collections are baked into the logic of the resolver. Everything will still be declarative, statically analyzable, and extensible.

I plan to post this proposal as a fresh RFC in the very near future. I'll try to address many of the points raised here in the new RFC. But if I miss any, and they still apply, please re-raise them in the new PR.

[stefanpenner]

@dgeb thanks for stewarding this process.

[dgeb]

Closing in favor of #143