Continuing the discussion from: http://www.ageofascent.com/reduce-node_modules-recursion-long-paths-asp-net-5/
Reduce node_modules recursion and long paths for ASP.NET 5
Trim your node_modules repetition by over 60%
However, if it was a runtime cost because we were running a node.js or io.js server; it would certainly be something I’d want to look at more closely!
For what it’s worth; in Node.js, requires are cached indefinitely. That means that the deduplication might slightly increase startup time (as it has to traverse to parent folders more often to find a dependency), but this is not going to have any performance effects on runtime.
A bigger concern is broken modules that rely on keeping state in a module - that’s generally a bad idea, as deduplication will cause possible conflicts and module state pollution (due to the module cache). In NPM 2 this was only occasionally a problem - you’d have to manually deduplicate - but in NPM 3 it’s far, far more likely to occur.
Realistically, though, that’s a module bug when it happens, and should be filed as such on the offending module…
Was more thinking of runtime performance of using 280 modules when you think you are using 20 as that’s a lot more code that is running. Although its more of a metrics scenario; if its fast enough; then convenience is more important.
If this is the case then you’d have a lot of issues serving concurrent requests?
It doesn’t matter, really. In Node.js, modules are effectively free - they are only modules at the point of require (and in the require cache), and beyond that they are just values like everything else. The amount of modules doesn’t matter, nor does the amount of values exported by them - the only thing that matters performance-wise is the amount of operations you carry out.
There may be a RAM penalty when using lots of modules, but this is so insignificant (even in big applications) that it’s even really worth consideration, especially as it’s static, and thus doesn’t scale with load. We’re talking megabytes here.
How so? The general approach in Javascript is to rely on closures for passing around state, which means you can share state wherever it is needed - you just put it in a different place. The fundamental point is that the scope of the module itself (ie. outside of any ‘constructor functions’, for example) shouldn’t contain state, as that makes it effectively global state due to the require cache.
And if two modules depend on the same sub-dependency, but each think that they will have their own copy and thus put state into that copy… then unknowingly, they will be sharing the sub-dependency, and thus affecting each other’s module state.
Never store state in modules, always keep state in closures at the appropriate level, and you should be fine. This is made a lot easier by using some functional constructs, for example.
EDIT: An example of ‘module state’, which is bad:
someModule = require("some-module");
someModule.thing = "cake";
This means that someModule.thing will now contain cake everywhere where the same module copy is used.
That kind of dependency redundancy is definitely an issue in Windows due to the ridiculously long filenames, and it’s an issue overall… size, etc. Good that there’s progress on that front. (Hate that you keep messing up “it’s” vs “its”, though… flees)
This is not a new feature, the only thing that’s new here is that npm v3 does this by default. You can use npm dedupe to do the same thing in older versions of npm.
Doesn’t npm dedupe behaviour change for npm3 also?
npm dedupe now flattens the tree in addition to deduping.
I’m a relative newbie to npm, so I could be wrong.
Hmm you could be right. I always thought dedupe flattened the tree, but maybe it doesn’t and my life is a lie. If it doesn’t flatten the tree, how does it dedupe modules?
Having a huge deep tree of modules was one of the worst decisions the developers of npm made. Literally every other package manager gets this right and does proper version resolution when you install packages, rather than npm’s approach. I’m glad they’re fixing it in npm 3.