Add project hash information to nx print-affected output
#22623
Replies: 34 comments 3 replies
-
|
This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. |
Beta Was this translation helpful? Give feedback.
-
|
I would love to have this to improve our CI pipeline as well - would save a lot of homegrown hacks |
Beta Was this translation helpful? Give feedback.
-
|
There is a bit of a workaround for this which is to run {
"tasks": [
{
"id": "lib-name:build",
"overrides": {},
"target": {
"project": "lib-name",
"target": "build"
},
"command": "yarn nx build lib-name",
"outputs": [
"dist/libs/lib-name"
],
"hash": "3128ba521cbe508342017bff3a2429d3bfc00cb6d17269b2078209bd17992108"
},
...etcI use this in combination with other |
Beta Was this translation helpful? Give feedback.
-
|
Ahh awesome, thank you so much for this - this will certainly do for now! |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately this workaround was removed in 67e78dc#diff-715d57e669c659f3ed1b596f901efab8bb79d61e16149eb088768110af4eed68. @vsavkin is there a different workaround that we might be able to use instead? |
Beta Was this translation helpful? Give feedback.
-
|
This is sad news indeed, I put a lot of work into rewriting our CI to use this workaround. @vsavkin are you able to shed some light on this? Is there another workaround or an official implementation being worked on? |
Beta Was this translation helpful? Give feedback.
-
|
I spent some time during a hackday to see if I could get something in our case to get back the hash on a per app/per lib case. I did dug into the diff mentioned by @zakhenry, I also looked into "NX plugins and devkit", "NX plugins", "Creating Custom Executors", the Plugins and custom executors won't be able to add something like/modify the I also tried a dummy approach of using NX through a node script like this: but unless I copy the whole I'm also confused by the commit that @zakhenry mentioned. This commit is called "feat(core): add support for custom hashers" so I'd expect the ability to customize the hashing strategy from a consumer point of view but maybe that's not it. Anyway from here I've got 2 things to highlight:
Thanks for any help on this 🙏 We're now stuck to nx 12.2.0 till we solve it as our CI pipeline relies on the hashes 🙏 |
Beta Was this translation helpful? Give feedback.
-
|
Any idea on this? Any guidance on this? I get 1 day per month to do what I want (hackday). As I said, I'm motivated to spend an entire day on this tomorrow if needed. If someone from NX could just help me out a little and give me a direction that'd be much appreciated 🙏 |
Beta Was this translation helpful? Give feedback.
-
|
If there isn't any plans to support exposing the hash again, which would be super sad, then maybe it is enough to use the Hasher in a custom script by importing it from
e.g. something like import { Hasher } from '@nrwl/workspace/src/core/hasher/hasher';
const hasher = new Hasher(projectGraph, nxConfig, {});
const hash = await hasher.hashTaskWithDepsAndContext(task); |
Beta Was this translation helpful? Give feedback.
-
|
This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. |
Beta Was this translation helpful? Give feedback.
-
|
Would still be nice to have. So bump for the stale label :) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the infos provided by everybody. If I'm not mistaken, this should show the hash for all projects, because compared to the first commit ever, they're all affected. |
Beta Was this translation helpful? Give feedback.
-
|
@Rijak0 which version of nx are you using? I think this issue slightly diverted from "It would be nice nice to have the hash for all projects" to "It'd be nice to have any hash at all" as they got removed completely from the output. This issue has been going on for a while so I may have lost track a bit and have my thoughts confused but at least I think that's the case. So I don't understand how you'd get a result with a hash in your first command? I tried it in our project and here's one object from the returned array: and there's no hash in here |
Beta Was this translation helpful? Give feedback.
-
|
Howdy @maxime1992! "Works on my machine" :D prints out Interesting that it doesn't work for you. |
Beta Was this translation helpful? Give feedback.
-
|
Still working for me with v 15.2.1:
|
Beta Was this translation helpful? Give feedback.
-
|
Bumping this up. I need to know each hash to remove old caches and keep caches folder as small as possible. PS: Although the solution above works, I've found a slightly better way of getting hashes. All such solutions are based on the same trick: we need Say you have defined your named inputs like this: Now if you say your global file changed, it will think that everything is affected and provide you with all the current hashes: Output:
|
Beta Was this translation helpful? Give feedback.
-
|
Seems like with nx v16 some logic to the hashing changed. The hash now looks quite different, e.g. Also looks like This is kind of a bummer, the hash really enabled some good automations and cicd options. I'll do some more digging and I'll try to find out what changed in between v15/16. |
Beta Was this translation helpful? Give feedback.
-
|
This should really be part of a public and stable API IMO. It's been working on and off too often. |
Beta Was this translation helpful? Give feedback.
-
|
+1… that is a core functionality, our whole CI/CD is also depending on it.
We‘re calculating affected differences with external systems like Docker,
Nuget, npm, for the build and deployment.
Also e.g. finding out which projects need to be re-tested…
…On Fri 1. Sep 2023 at 14:00, Maxime ***@***.***> wrote:
This should really be part of a public and stable API IMO. It's been
working on and off too often.
If any maintainer read this, could you please consider having the hash
somehow part of a stable API for the CLI please?
—
Reply to this email directly, view it on GitHub
<#3292 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABZMCMG5ZLEEE2V6VELIZC3XYHE55ANCNFSM4OVQDV7A>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Exactly.
We do the same. We take advantage of the fact that Gitlab is capable of having a first stage to generate a gitlab ci yaml file that is then run as the pipeline (aka dynamic pipelines on top of my head). So we have our first pipeline running as a deno script and checking the result of print-affected to analyse what's changed, and from there we generate all the pipeline (lint, test, e2e tests, docker builds, deployment etc) dynamically. While the hashes are not necessary to do that, they are for further optimisations. For example, it'd be nice to save the docker build of projects by their hash. This way we could query in our first pipeline the registry to see if a tag already exist for a given hash, if so, completely skip a build because it's been done and whatever changed since, didn't affect the files for that project. |
Beta Was this translation helpful? Give feedback.
-
|
@vsavkin @AgentEnder do you have any opinions on this? Some kind of stable api or cli to consistently get the hashes would be extremely helpful. Even adding it to the graph commands would be great (when outputting to json or similar) would work. Would gladly help with implementing this, if you have a vision for what the implementation should approx. look like 🙏 |
Beta Was this translation helpful? Give feedback.
-
|
A little hacky, but seems to be working: returns for a new playground nx workspace: |
Beta Was this translation helpful? Give feedback.
-
For: +1 to this being provided as standard output from one of the affected/graph commands. |
Beta Was this translation helpful? Give feedback.
-
|
Stuck on v17.1 for now. @vsavkin Hopefully this can get some attention soon to unblock us - and I suspect many others. |
Beta Was this translation helpful? Give feedback.
-
|
@dsschneidermann I am still on v17.1 due to other issues preventing me from an upgrade, but I adapted your example today to using the NX Deamon APIs, maybe this might work more stable with v17.2 and solves the issues you are seeing? import { createProjectGraphAsync, ProjectGraph } from '@nx/devkit';
import { DaemonClient } from 'nx/src/daemon/client/client';
import { DaemonBasedTaskHasher } from 'nx/src/hasher/task-hasher';
import { createTaskGraph, mapTargetDefaultsToDependencies } from 'nx/src/tasks-runner/create-task-graph';
import { TargetDependencies, readNxJson } from 'nx/src/config/nx-json';
const uniq = <I,>(arr: I[]): I[] => [...new Set(arr))] as I[];
const resolveTargetHashes = async ({
target,
graph,
dependencies,
hasher,
}: {
target: string;
graph: ProjectGraph;
dependencies: TargetDependencies;
hasher: DaemonBasedTaskHasher;
}) => {
const projectNames = Object.values(graph.nodes)
.filter(({ data }) => !!data.targets?.[target])
.map((p) => p.name);
const taskGraph = createTaskGraph(graph, dependencies, projectNames, [target], undefined, {}, false);
const tasks = Object.values(taskGraph.tasks);
const hashes = await hasher.hashTasks(tasks, taskGraph, process.env);
return hashes.map((res) => res.value);
};
async function exec() {
const cwd = process.cwd();
const nx = readNxJson(cwd);
const graph = await createProjectGraphAsync({ exitOnError: true, resetDaemonClient: true });
const daemon = new DaemonClient(nx);
const hasher = new DaemonBasedTaskHasher(daemon, {});
const dependencies = mapTargetDefaultsToDependencies(nx.targetDefaults);
// Adapt the following line to your own likings, I automatically resolve all targets with caching enabled based on the targetDefaults.
const targets = Object.entries(nx.targetDefaults ?? {})
.filter(([target, config]) => config.cache === true)
.map(([target]) => target);
const hashes = await Promise.all(
targets.map((target) => resolveTargetHashes({ target, graph, dependencies, hasher })),
).then((results) => uniq(results.flat()));
console.log(hashes);
}
exec()
.then(() => {
process.exit(0);
})
.catch((err) => {
console.error(err);
process.exit(1);
}); |
Beta Was this translation helpful? Give feedback.
-
|
Yes, functionality like this is truly needed. I hope someone on the team becomes aware of that. Ideally, we should be able to obtain the hash via a simple CLI command like:
This way, nx would be much more powerful. Especially in the context of AWS, using hashes is crucial, as there isn't any other good indicator for checking whether or not a resource needs to be updated AND having some kind of indiciator about the deployments current build. Potential time savings could be enormous. For now, we have to use simple commit hashes, sadly. From an infrastructure/deployment standpoint, this is more or less THE potential killer feature of why people would choose NX instead of Bazel or other solutions. However, as of now, NX simply doesn't provide this core functionality to fully utilize its incremental build system. I would greatly appreciate it if the team could address this issue. |
Beta Was this translation helpful? Give feedback.
-
|
I've updated the code of @Inkdpixels and made a little CLI tool out of it (also works with 17.2.8), which allows you to specifically filter by project and target. Output is the requested hash value. import { createProjectGraphAsync, ProjectGraph } from "@nx/devkit"
import { DaemonClient } from "nx/src/daemon/client/client"
import { DaemonBasedTaskHasher } from "nx/src/hasher/task-hasher"
import { createTaskGraph, mapTargetDefaultsToDependencies } from "nx/src/tasks-runner/create-task-graph"
import { TargetDependencies, readNxJson } from "nx/src/config/nx-json"
import * as yargs from "yargs"
const argv = yargs
.option("project", {
alias: "p",
description: "Filter result by project.",
type: "string",
})
.option("target", {
alias: "t",
description: "Filter result by target.",
type: "string",
})
.help()
.alias("help", "h")
.parseSync()
const resolveTargetHashes = async ({
target,
graph,
dependencies,
hasher,
}: {
target: string
graph: ProjectGraph
dependencies: TargetDependencies
hasher: DaemonBasedTaskHasher
}) => {
const projectNames = Object.values(graph.nodes)
.filter(({ data }) => !!data.targets?.[target])
.map((p) => p.name)
const taskGraph = createTaskGraph(graph, dependencies, projectNames, [target], undefined, {}, false)
const tasks = Object.values(taskGraph.tasks)
const result = await Promise.all(
tasks.map((task) =>
hasher.hashTask(task, taskGraph, process.env).then((hash) => ({
id: task.id,
target: task.target,
hash: hash.value,
})),
),
)
return result
}
async function exec() {
const cwd = process.cwd()
const nx = readNxJson(cwd)
const graph = await createProjectGraphAsync({ exitOnError: true, resetDaemonClient: true })
const daemon = new DaemonClient(nx)
const hasher = new DaemonBasedTaskHasher(daemon, {})
const dependencies = mapTargetDefaultsToDependencies(nx.targetDefaults)
const targets = Object.entries(nx.targetDefaults ?? {})
.filter(([target]) => target === argv.target)
.map(([target]) => target)
if (!targets.length) {
console.error(`Project '${argv.project}' with target '${argv.target}' not found!`)
process.exit(1)
}
const targetsWithHashes = await Promise.all(targets.map((target) => resolveTargetHashes({ target, graph, dependencies, hasher }))).then((results) => results.flat())
const filteredTarget = targetsWithHashes.filter((target) => target.target.project === argv.project)
if (!filteredTarget.length) {
console.error(`Project '${argv.project}' with target '${argv.target}' not found!`)
process.exit(1)
}
if (!filteredTarget[0].hash) {
console.error(`No hash for target ${argv.target} of project ${argv.project} found`)
process.exit(1)
}
console.log(filteredTarget[0].hash)
}
exec()
.then(() => {
process.exit(0)
})
.catch((err) => {
console.error(err)
process.exit(1)
})Usage:
will result in:
|
Beta Was this translation helpful? Give feedback.
-
|
Hey all, Nx cannot generate the correct hash for every task without actually running the command. The tasks are hashed right before they executed in the middle of overall task execution. This is because tasks can depend on the outputs of tasks they depend on via https://nx.dev/reference/inputs#outputs-of-dependent-tasks. The same is true for custom hashers because Nx does not know what information custom hashers are hashing.. so it could be information which changes while other tasks execute. We'll need to discuss more if this limitation is a deal breaker for this. Nx could provide the hashes of only tasks which it knows for sure should be accurate. But it wouldn't be a definitive list of hashes which Nx will run. There are cases where none of the tasks in the command can actually be hashed upfront. So I wouldn't build anything on top of it such as spawning CI agents. I think this discussion is better held at a higher level. What is the desired outcome of having the hashes? If we could discuss that, we could see if this information would suffice or if there is a better way to go about it in a way that is actionable. |
Beta Was this translation helpful? Give feedback.
-
|
The use case for us is to know that a project has changed, and the hash is used to determine if we have seen it before (e.g. a developer pushed a commit that changes no files but amends the commit message, we've seen this project change before and don't need to rerun CI on it), or more usefully, if a developer makes a core lib change in one commit, we rebuild all dependents in that CI pipeline, but subsequent changes that only affect one of the projects depending on that original change should only build the dependent project. |
Beta Was this translation helpful? Give feedback.
-
|
Not rerunning CI at all is... not really something that Nx can tell you 100% of the time. But rerunning CI when the same thing has been run already should re-utilize the cached runs from before. My suggestion is to use Even if Nx were to provide the hashes... they would not be accurate enough to guarantee that nothing really has changed without running CI at all.
|
Beta Was this translation helpful? Give feedback.
-
|
One of the use cases for which I utilize the hash is fully automated CD resource naming. For instance, tagging a Docker image with the NX-generated hash significantly reduces deployment time. Most deployment systems incorporate some form of internal caching functionality to check whether the latest deployed software piece is identical to the new one. However, by naming it differently with each new deployment, it will always deploy as if it's a newer version, regardless of any changes made. Thus, it allows for more precise deployments. Since caching deployment via NX isn't advisable, utilizing the built-in caching functionalities of, for example, AWS CloudFormation, ECS, etc., becomes possible. Especially in the context of the Cloud, where Git commit hashes are typically used for naming/tagging resources, the importance of being able to name by hash becomes evident. It simply aids in reducing deployment time. One might argue that using just one name could achieve the same result. Technically, yes, but practically, no. Several systems and services lack any form of integrated version history. Having to rollback every change based solely on the commit history due to a bug would be problematic, especially with larger deployments that introduce multiple new features. Furthermore, achieving targeted rollbacks without additional means of identifying unique software deployment parts would be unnecessarily complex and complicated. I'm certain there are several more reasons, but overall, these are my personal points for why I prefer the ability to use the hash. |
Beta Was this translation helpful? Give feedback.
-
|
Seems like as if the hasher functionality with v19 doesn't work as expected anymore and instead got really slow performance wise, so that the script provided by @Inkdpixels doesn't work anymore. Takes like three minutes, which makes it practically unuseable. For now I'll stay on v18. @FrozenPandaz is there a reason for the function |
Beta Was this translation helpful? Give feedback.
-
Description
I'd like to see a set of fields added to the print affected json output, something like
{ "projects": [...], "projectGraph": {...}, "hashes": { "my-app": { "current": "4b669d95627bf6724502d7b4ff6f41d3654c50ba9e3b1cf31e72c138370423f8", "previous": "0bdb20fbd1788957302f4cf356ff0449ee3c2d10960814f6c34fd399fa64f5d3" } } }This hash key should include all projects, not just the affected ones.
Motivation
I use nx print-affected in order to dynamically generate a CI pipeline which works amazingly well, however I have a need to determine the current hash of an unchanged project so that I can fetch the docker container
Suggested Implementaion
I assume that the affected projects is determined by some hashing already, so it must be feasible to add this into the affected output
Alternate Implementations
This could be a completely different command line tool if needed.
Beta Was this translation helpful? Give feedback.
All reactions