Base.include() reports "LoadError: syntax: expression too large"

[wupeifan]

I have a function that creates an array and fills each element of that array. The system is large as there are about 400k lines. And it encounters an error "LoadError: syntax: expression too large" when I try to "Base.include()" it. I am not sure whether this is a bug (or it is a feature) and whether there are ways to get around it.
Since I have a function to include, I cannot really import each line one by one. It has to be as a whole.
Attached here is the file that I try to "Base.include()" (after unzipping you'll see a Julia file)
And here goes the error message:

ERROR: LoadError: syntax: expression too large
Stacktrace:
 [1] top-level scope at C:\(I omit the path here)\Model_4753687013.jl:1
 [2] include(::Module, ::String) at .\Base.jl:377

Model_4753687013.zip

[sgaure]

That's an interesting one. It seems you reach some complexity limit for either include() or function sizes. To get around it you could compact the code, e.g. instead of

var"##MTIIPVar#284"[1] = 0
var"##MTIIPVar#284"[2] = 0
var"##MTIIPVar#284"[3] = 0
... 
var"##MTIIPVar#284"[173] = 0

you could write

var"##MTIIPVar#284"[1:173] .= 0

or zero the entire array with
var"##MTIIPVar#284"[:] .= 0
before you fill in the non-zero entries (which are relatively few).

Of your 424925 lines, 424087 ends in "= 0", so there could be some space to save there.

[KristofferC]

Clearly, this is the expanded from of some procedure to generate these arrays. But why have you expanded / unrolled that procedure into all of its array assignments? Just run it when you need the array. Or alternatively, write the array to disc if you want to cache it's creation. Encoding big data into source code like this is terribly inefficient and there will always be some upper bound on file size you can run up against (which will be much lower than what e.g. reading the data from disc will be).

[mcabbott]

BTW the limit seems to have changed recently (causing JuliaMath/Sobol.jl#27) so being just under it on 1.4 doesn't guarantee you'll be safe in future.

[JeffBezanson]

Ah, interesting. We didn't deliberately change the limit, but rather this is an unintended consequence of #35243, where there is now sometimes an extra julia->scheme conversion step.

[wupeifan]

Yep, we are rewriting part of the code to fill the non-zero entries instead of each element of the array. That would solve our problem, and thanks for the suggestions above.
I'm still interested in the upper limit of the function size.

Of your 424925 lines, 424087 ends in "= 0", so there could be some space to save there.

Yes, indeed, we have a sparse version to compare with this dense one.

Just run it when you need the array. Or alternatively, write the array to disc if you want to cache it's creation.

Notice that this is a function that returns an array. We want to cache the function that we created because the creation takes a bit while.

[mcabbott]

Perhaps you can copy what Sobol did (above), which loads a CSV file at during precompilation:

https://github.com/stevengj/Sobol.jl/blob/master/src/soboldata.jl#L43

[wupeifan]

That's also a good alternative. Thanks!
I think I'll keep track of the limit issue that related to #35243 and to see whether it will be adjusted soon (I am too lazy)...

[laborg]

On master the attached model can be included without error. Probably because #36612 and/or JuliaSyntax.jl

julia> filesize("Model_4753687013.jl")
23707392

julia> include("Model_4753687013.jl")
Main.Model_4753687013