Why are XML and INI treated differently in statistics?

[biqqles]

The readme states:

Linguist starts by going through all the files in a repository and excludes all files that [...] are defined as data [...] whilst taking into account any overrides.

I am trying to determine why XML and INI files are treated differently in language statistics. The quoted sentence says that data files are excluded, yet XML shows up in language statistics (for example, for this repository) even without any overrides.

At the same time, INI, another data filetype, does not show up. This is expected because data files should be excluded. However, even when I tried adding a .gitattributes containing *.ini linguist-detectable=true, INI does not show up in the repository statistics. I suspected this would be the case because there are zero repositories marked as having INI as their language.

So what is going on here? I'm not sure whether this is a bug, or intended behaviour. Firstly, why does XML show up in statistics even though it is of type data? And secondly, why does INI not show up in statistics even when overridden?

[lildude]

Firstly, why does XML show up in statistics even though it is of type data?

This is happening because that repo was last touched over eight years ago. This predates XML being marked as a data language six years ago. Repos are only analysed when changes are pushed to them.

And secondly, why does INI not show up in statistics even when overridden?

It should do. There are a few reasons why it may not, in order of likelihood:

• the background job that performs the analysis still hadn't run when you last checked
• the override has a mistake in it or the path isn't quite right
• the file in question is in a location affected by the other ignored file, like docs or vendored so need additional overrides.
• the background job that performs the analysis hit a problem causing it to fail

Show us the repo with the override in place where you're seeing this and we can take a look.

[biqqles]

Thanks for your answer. I am going to double check my overrides (I just realised the files are probably getting marked as vendored code) and then get back to you.

Also, part for my thesis for INI "not working" was that according to search there was not a single repository marked as having it as a language. Well, incredibly weirdly, after posting this yesterday I repeated my search on mobile and found this repo. This morning, the search results on mobile are back to being empty just like on desktop. This part of the equation then is presumably GitHub's search being broken and nothing to do with Linguist.

[biqqles]

The repo is here. I only added .gitattributes a few hours ago so I'm not sure if it has been analysed yet but you might be able to tell already whether there are any glaring errors or it looks OK.

[lildude]

Yup, looks like a combination of these two:

• the override has a mistake in it or the path isn't quite right
• the file in question is in a location affected by the other ignored file, like docs or vendored so need additional overrides

$ git clone -q [email protected]:biqqles/aurora.git
$ cd aurora/
$ cat .gitattributes
dist/* linguist-vendored=false
*.ini linguist-detectable=true
$ git check-attr linguist-detectable cursors/dist/DATA/mouse.ini
cursors/dist/DATA/mouse.ini: linguist-detectable: true
$ git check-attr linguist-vendored cursors/dist/DATA/mouse.ini
cursors/dist/DATA/mouse.ini: linguist-vendored: unspecified
$

Your .ini files fall into a vendored directory so you need the vendor negation override as well as the detectable override, however the path you've specified to negate the vendoring is too specific (it only applies to files directly in the dist directory off the root of the repo) and needs expanding, either to be more specific or by adding more wildcarding because * doesn't recurse. See the discussion at #5175 (comment) for links to docs etc.

The "more wildcarding" approach of */dist/** linguist-vendored=false (or */dist/** -linguist-vendored) will do the trick:

$ cat .gitattributes
*/dist/** linguist-vendored=false
*.ini linguist-detectable=true
$ git check-attr linguist-vendored cursors/dist/DATA/mouse.ini
cursors/dist/DATA/mouse.ini: linguist-vendored: false
$ github-linguist . --breakdown
100.00% INI

INI:
cursors/dist/DATA/mouse.ini
fonts/dist/DATA/FONTS/fonts.ini
fonts/dist/DATA/FONTS/rich_fonts.ini
$

Alternatively, you can be more specific if you want using cursors/dist/DATA/* linguist-vendored=false:

$ cat .gitattributes
cursors/dist/DATA/* linguist-vendored=false
*.ini linguist-detectable=true
$ git check-attr linguist-vendored cursors/dist/DATA/mouse.ini
cursors/dist/DATA/mouse.ini: linguist-vendored: false
$ github-linguist . --breakdown
100.00% INI

INI:
cursors/dist/DATA/mouse.ini

$

I've deliberately demonstrated using git check-attr to check the attributes to take Linguist out of the picture before confirming how this applies to Linguist.

[biqqles]

Aha, thank you! I did wonder whether * would recurse but I decided to just closely follow the examples in the readme. git check-attr is actually really useful, I will bear that in mind in future. Thanks again.