Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--stream prints newlines as \n #994

Closed
FezVrasta opened this issue Aug 29, 2017 · 18 comments
Closed

--stream prints newlines as \n #994

FezVrasta opened this issue Aug 29, 2017 · 18 comments

Comments

@FezVrasta
Copy link
Contributor

Expected Behavior

I run my tests with lerna run --stream test, the packages tests are simple Jest tests.
The output printed by Lerna doesn't properly renders the newlines, for instance, this is what I get:

image

Possible Solution

I'm not sure, but it looks like Lerna does something to escape the special characters and forgets to restore them before it prints.

Steps to Reproduce (for bugs)

  1. create a package/something with inside its package.json a Jest test command
  2. run lerna run --stream test in the root of the project
lerna.json

{
  "lerna": "2.1.1",
  "packages": [
    "packages/*"
  ],
  "version": "independent",
  "npmClient": "yarn"
}

lerna-debug.log

<!-- If you have a `lerna-debug.log` available, please paste it here -->
<!-- Otherwise, feel free to delete this <details> block -->

Context

This bug makes it very difficult to read the logs, especially in CI

Your Environment

Executable Version
lerna --version 2.1.1
npm --version 5.3.0
yarn --version 1.0.0-20170815.1425
node --version 8.1.2
OS Version
macOS 10.12.6
@evocateur
Copy link
Member

Yikes, this is pretty bad.

Perhaps this console.log should be process.stdout.write()? And we can probably dispense with the fancy spread ...args and just call the single param message.

@ptshih
Copy link

ptshih commented Sep 5, 2017

This also happens with lerna run --parallel

@frankwallis
Copy link

I'm seeing this when running lerna run build without using --stream or --parallel. I'm using [email protected] and the build command runs webpack:

lerna ERR! execute         + 8 hidden modules
lerna ERR! execute
lerna ERR! execute     at Promise.all.then.arr (/usr/lib/node_modules/lerna/node_modules/execa/index.js:236:11)
lerna ERR! execute     at <anonymous>
lerna ERR! execute     at process._tickCallback (internal/process/next_tick.js:188:7)
lerna ERR! execute   stack: 'Error: Command failed: npm run build-dev\nclean-webpack-plugin: /mnt/c/work/cofco/mergerepos/monorepo3/packages/ca.ui.module.core/dist has been removed.\n\n[at-loader] Checking finished with 1 errors\nnpm ERR! code ELIFECYCLE\nnpm ERR! errno 2\nnpm ERR! [email protected] build-dev: `webpack --env.development`\nnpm ERR! Exit status 2\nnpm ERR! \nnpm ERR! Failed at the [email protected] build-dev script.\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /home/frank/.npm/_logs/2017-09-15T09_37_36_206Z-debug.log\n\n> [email protected] build-dev /mnt/c/work/cofco/mergerepos/monorepo3/packages/ca.ui.module.core\n> webpack --env.development\n\nNot using source maps at all for development\nNot using full source maps for dependencies\n\n[at-loader] Using [email protected] from typescript and "tsconfig.json" from /mnt/c/work/cofco/mergerepos/monorepo3/packages/ca.ui.module.core/tsconfig.json.\n\n\n[at-loader] Checking started in a separate process...\nHash: 9bf04aa8f24862a52d3d\nVersion: webpack 3.5.6\nTime: 30474ms\n
                                  Asset       Size  Chunks                    Chunk Names\n           component-assets/DejaVuSans-Bold.ttf     704 kB          [emitted]  [big]  \n             component-assets/columnGroupL1.svg  953 bytes
       [emitted]         \n             component-assets/columnGroupL3.svg    1.05 kB          [emitted]         \n             component-assets/cofco-sprites.png  288 bytes          [emitted]         \n                    component-assets/sprite.png    13.3 kB          [emitted]         \n                   component-assets/loading.gif    2.74 kB          [emitted]         \n             component-assets/loading-image.gif    6.54 kB          [emitted]         \n
component-assets/indeterminate.gif    2.12 kB          [emitted]         \n                  component-assets/slider-h.gif   63 bytes          [emitted]         \n                  component-assets/slider-v.gif   63 bytes          [emitted]
         \n              component-assets/imagebrowser.png    6.21 kB          [emitted]         \n                component-assets/sprite_kpi.png  787 bytes          [emitted]         \n                 component-assets/sprite_2x.png    33.7 kB          [emitted]         \n             component-assets/sprite_kpi_2x.png    1.47 kB          [emitted]         \n                   component-assets/markers.png  988 bytes          [emitted]         \n                component-assets/markers_2x.png    2.17 kB          [emitted]         \n              component-assets/transparency.png    1.83 kB          [emitted]         \n             component-assets/KendoUIGlyphs.eot    22.5 kB          [emitted]         \n
         component-assets/transtexture.png  203 bytes          [emitted]         \n                       component-assets/hue.png    1.71 kB          [emitted]         \n             component-assets/KendoUIGlyphs.ttf    22.3 kB          [emitted]         \n            component-assets/KendoUIGlyphs.woff    22.4 kB          [emitted]         \n             component-assets/KendoUIGlyphs.svg    61.6 kB          [emitted]         \n                component-assets/DejaVuSans.ttf     756 kB          [emitted]  [big]  \n             component-assets/columnGroupL2.svg    1.04 kB          [emitted]         \n        component-assets/DejaVuSans-Oblique.ttf     634 kB          [emitted]  [big]  \n    component-assets/DejaVuSans-BoldOblique.ttf     642 kB          [emitted]  [big]  \n               component-assets/DejaVuSerif.ttf     380 kB          [emitted]  [big]  \n          component-assets/DejaVuSerif-Bold.ttf     356 kB          [emitted]  [big]  \n        component-assets/DejaVuSerif-Italic.ttf     346 kB          [emitted]  [big]  \n    component-assets/DejaVuSerif-BoldItalic.ttf     347 kB          [emitted]  [big]  \n            component-assets/DejaVuSansMono.ttf     340 kB
       [emitted]  [big]  \n       component-assets/DejaVuSansMono-Bold.ttf     332 kB          [emitted]  [big]  \n    component-assets/DejaVuSansMono-Oblique.ttf     251 kB          [emitted]  [big]  \ncomponent-assets/DejaVuSansMono-BoldOblique.ttf     253 kB          [emitted]  [big]  \n     component-assets/MaterialIcons-Regular.eot     143 kB          [emitted]         \n   component-assets/MaterialIcons-Regular.woff2    44.3 kB          [emitted]         \n    component-assets/MaterialIcons-Regular.woff    57.6 kB          [emitted]         \n     component-assets/MaterialIcons-Regular.ttf     128 kB          [emitted]         \n        component-assets/AvenirLTStd-Medium.otf    28.1 kB          [emitted]
         \n         component-assets/AvenirLTStd-Heavy.otf    28.5 kB          [emitted]         \n          component-assets/AvenirLTStd-Book.otf    28.3 kB          [emitted]         \n         component-assets/AvenirLTStd-Roman.otf
28 kB          [emitted]         \n                      component-assets/link.png  389 bytes          [emitted]         \n          component-assets/cofco-logo-19x19.png    1.15 kB          [emitted]         \n                           ca.ui.module.core.js    12.7 MB       0  [emitted]  [big]  main\n                          ca.ui.module.core.css     922 kB       0  [emitted]  [big]  main\n                                     theme.less    10.9 kB          [emitted]

@charlesbjohnson
Copy link

charlesbjohnson commented Dec 25, 2017

I also ran into this, but it seems to be an issue primarily with jest. Relevant issues:

jest's BaseReporter writes output to stderr instead of stdout. See here.

Since lerna run collapses newlines for stderr via strong-log-transformer, we end up with escaped newlines in the output.

const prefixedStderr = logTransformer({ tag: `${color(prefix)}:`, mergeMultiline: true });

I'm not entirely sure why this is jest's default behavior since it doesn't seem to be consistent with other test runners (notably mocha). As such, I think the fix probably belongs in jest and not in lerna.

Removing mergeMultiline in lerna might not break anything (--parallel?), but it seems like it could also have larger implications since lerna run is not limited to running tests.

There is a very simple workaround to perform the redirection to stdout in bash, though this still probably won't work well with CI tools that read from stderr to determine build failures.

// /repo/packages/sub/package.json
"scripts": {
  "test": "jest 2>&1"
}
// /repo/package.json
"scripts": {
  "test": "lerna run test"
}

@A77AY
Copy link

A77AY commented Jan 8, 2018

@FezVrasta,

I think you can temporarily use exec. It does not have such a problem:

lerna exec --scope <scope> --stream -- npm run <scriptName>

You can also wrap scripts in package.json (for example with concurrently):

"scripts": {
    "start": "concurrently: \"<script>\""
}

or with npm-run-all:

"scripts": {
    "start": "npm-run-all --print-name --serial <scriptName>",
    "scriptName": "<script>"
}

@alexeyraspopov
Copy link

Removing mergeMultiline in lerna might not break anything (--parallel?), but it seems like it could also have larger implications since lerna run is not limited to running tests.

Could we possibly discuss this option? What are the possible implications?

@eritikass
Copy link

eritikass commented Mar 26, 2018

Removing mergeMultiline in lerna might not break anything (--parallel?), but it seems like it could also have larger implications since lerna run is not limited to running tests.

Could we possibly discuss this option? What are the possible implications?

2'nd that - jest is hugely popular and this is making usage of jest very problematic. Also by the comments for jest developers, its working as excepted - so not much hope they would change it.

Perhaps some env var or flag that can be used to disable mergeMultiline?

when i tested and removed mergeMultiline from

const prefixedStderr = logTransformer({ tag: `${color(prefix)}:`, mergeMultiline: true });

it did work nicely with jest (happy to provide PR if any insight about accepted solution can be provided)

@evocateur
Copy link
Member

evocateur commented Mar 26, 2018 via email

@eritikass
Copy link

Our use-case, we did merge several heavily connected JS projects into one for more simple management, so we do not have to make multiple PR into different projects same time etc...

lerna did help a lot to manage this somewhat ad-hock monorepo. bootstrap to npm install all projects , run to run commands in all projects lerna run lint / lerna run start / lerna run build / ... did not work that well for testing.

@alexeyraspopov
Copy link

@evocateur, yarn lerna run test --since master on a branch allows you to run tests on the only packages that has changes.

@evocateur
Copy link
Member

evocateur commented Mar 26, 2018 via email

@alexeyraspopov
Copy link

@evocateur, by "transitive dependencies" do you mean sibling packages that appeared to be not tested with the command (since weren't changed)? Having "always green master" allows not to run all the tests, since packages weren't modified.

Brilliant strategy to save a few seconds.

Imagine a dozen of client-side apps in a single repo, each of them with a set of e2e tests. This takes way longer than a few seconds. Especially, on CI server, where a few concurrent PRs are running.

@morgs32
Copy link

morgs32 commented Apr 4, 2018

@evocateur
Why would tests run in a fraction of the time? Just curious
#994 (comment)

@lehni
Copy link

lehni commented Jul 2, 2018

@evocateur regarding your question here ☝️ :

I have a monorepo in which different packages have different .babelrc files, for backend and frontend / vue.js specific code. I failed to find a nice way so far to set up jest from root in an easy way so it could run all the tests at once and still respect the different .babelrc settings, so I am using it through lerna exec instead.

Do you know of an easy way to make this work from root? If not, then this would be an example of a reson for using lerna exec with Jest.

@evocateur
Copy link
Member

evocateur commented Jul 3, 2018 via email

@lehni
Copy link

lehni commented Jul 3, 2018

@evocateur I've tried that but to no avail. It looks like Jest projects aren't picking up the .babelrc files in the nested folders, and I couldn't figure out how to use transform to provide the same configuration.

@lehni
Copy link

lehni commented Jul 4, 2018

@evocateur I found a solution for nested .babelrc and Jest now, but I needed to write my own version of babel-jest to be able to do so. The explanation is here, in case it's of interest for others:

https://github.com/lehni/babel-jest-nested#example-with-lerna

https://stackoverflow.com/questions/50511308/jest-projects-configuration-does-not-read-babelrc-and-webpack-config-json-in/51174657#51174657

@lock
Copy link

lock bot commented Dec 27, 2018

This thread has been automatically locked because there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Dec 27, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants