Allow configuration from config files as well

[wackywendell]

For example, imagine you have a program foobar that has many configuration options. You want them to be available on the command-line, but you also want to allow users to write a .foobarrc (perhaps in TOML format) so that they can pre-specify their choices. Or maybe there might be multiple configuration files, one in the home directory ($HOME/.foobarrc), one in the current directory (./.foobarrc), and maybe even one specified on the command-line.

If this is outside the scope of this library, perhaps adding more ways to extract information from an App (such as get_names(&self) -> [&str], which would list all the names of the arguments defined) would allow someone else to build this.

I ask because I am building an application that could use something like this - ParView - and I can't find a good way to do this without having to write every config option in multiple places.

[WildCryptoFox]

Interesting problem... At least for now, I'm going to suggest Option's or_else to extract the fields from your secondary source. Here; we preference any arguments supplied.

[wackywendell]

What two lists? I don't understand what you're suggesting.

I would like to be able to allow config files with a limited set of valid configuration options. But... how do I provide those options to clap, and also to my TOML validator, without repeating myself, and while allowing for all the options clap allows for long arguments, short arguments, help, etc.?

[WildCryptoFox]

It would appear that you responded to an older message of mine, that I had deleted. Thoughts for my updated message?

[wackywendell]

I don't entirely understand your new suggestion.

Let's say I have thirty configuration options, with various clap features that I would like:

let app = App::new("myapp")
    .arg(Arg::with_name("foo")
        .short("f")
        .long("foo")
        .help("Sets a custom foo")
        .takes_value(true))
    .arg(Arg::with_name("bar")
        .help("Sets baz")
        .required(true)
        .index(1))
    .arg(Arg::with_name("baz")
        .short("b")
        .multiple(true)
        .help("Sets the bazness"))
        [...]

So I can create the app that way. Now, how do I get a list of options to pass to my TOML validator? I could have made it ahead of time, but that's cumbersome if I want to use features like .multiple or .short. AFAICT, I can't extract the list from app, as there is no method available to do so, and App::flags etc. are private. So... how do I tell the TOML validator what are and are not valid options, without basically writing every config option in two different places?

[sru]

I am not sure whether this is in the scope of this library or not. @kbknapp could comment on this.

Maybe you could use .validator() method on Arg, here is the doc.

[kbknapp]

@wackywendell

I see two possible actions for your question, so bare with me if one of them is way off track! It may also help if you show us an example of what a .foobarrc could look like.

Pre-Specifying Argument Values

Meaning, your application accepts -a some_value and the user leaves out that option because it's specified in a .foobarrc file somewhere. I.e. you are not using the TOML files to create valid arguments, only use valid arguments. If instead, you are trying to create valid arguments with the TOML files, skip down to the next section.

To use valid arguments you shouldn't need to pass any Arg stucts to your TOML validator because clap does the validating, the TOML parser just supplies the values for clap to parse. In essence, you're trying to merge what the user passes in env::args() and these TOML files to create something for clap to parse.

First, you could pre-check for your .foobarrc before calling any of the get_matches methods, and instead use a get_matches_from which would be a merged version of what was provided in .foobarrc and env::args(). The cumbersome part is doing the merging/filtering which would have to be done manually.

The next option is just to use default values using .unwrap_or(get_default_from_toml(".foobarrc") for the args which might have a default value, where get_default_from_toml() would search this TOML file and pull out any defaults. Again, the cumbersome part is having to write that function manually.

Finally, you could also re-parse a simulated arguments list if there are errors on parsing. But doing this would require all the potentially default arg values to be required (i.e. in order to cause an error when omitted).

If you're looking for a way to programmatically change Args within an App struct, your only options are to have separate let mut bindings for each, and re-create the App struct on a failed parse, or after reading a .foobarrc. Again this is kind of manual.

Getting A List of Valid Arguments from the App struct

If you're trying to create valid arguments from these TOML files, your options are a little more limited. In those cases I would almost exclusively recommend parsing those files FIRST to create an App struct, and add your static (i.e. known at development time) Args after.

If there is overlap, you almost certainly want it to error or panic, unless you truly want users to be able to change arguments at runtime...which seems dangerous! Adding arguments seems safe, but changing existing is ones is something I'd think hard about 😜

Having said that, I have it on the back burner to add a method to get a &mut Arg out of an App struct. This is somewhat complicated though due to how Args are parsed and such, which would pretty much require all of the internals of App to be re-checked because literally anything could have fundamentally changed about the Arg changing it into a new type of argument entirely.

The other option is to add some way to get a &Arg which would be far easier, but of less use unless you are dynamically generating your Args and don't actually know what they are at development time. If this is what you're looking for, let us know and I'll add a tracking issue to add it.

Because &Arg doesn't differentiate from various types of arguments, we have internal structs which do and we could also expose those various "builders"; FlagBuilder, OptBuilder, and PosBuilder but those aren't really meant for public consumption because they change from time to time.

Hopefully this helped a little. If not, please let us know! Thanks for taking the time to file this issue! 😄

[WildCryptoFox]

The next option is just to use default values using .unwrap_or(get_default_from_toml(".foobarrc") for the args which might have a default value, where get_default_from_toml() would search this TOML file and pull out any defaults. Again, the cumbersome part is having to write that function manually.

unwrap_or assumes there will be some value; you want or_else which is the inverse of and_then. The value may or may not also be in the toml.

[kbknapp]

Good point! I went through a few iterations of that comment and didn't
re-screen for changes or test.

On Fri, Sep 11, 2015, 11:21 PM James McGlashan [email protected]
wrote:

The next option is just to use default values using .
unwrap_or(get_default_from_toml(".foobarrc") for the args which might
have a default value, where get_default_from_toml() would search this
TOML file and pull out any defaults. Again, the cumbersome part is having
to write that function manually.

unwrap_or assumes there will be some value; you want or_else which is the
inverse of and_then. The value may or may not also be in the toml.

—
Reply to this email directly or view it on GitHub
#251 (comment).

[wackywendell]

Thanks for the thoughtful and thorough response!

I was thinking of option 1: the final user of the application creates a .foobarrc so that they don't have to write --foo --bar 8 on the command-line every single time they run the program.

Your suggestions on how to merge TOML config and clap matches make sense to me. However, there is one thing that this leaves out. clap does a very good job of telling the user when they put in an incorrect argument, and even suggests differences; I don't see a good way to do that with a TOML file without having to write the app settings in two places.

For example, let's imagine we have this app:

let app = App::new("myapp")
    .arg(Arg::with_name("foo")
        .short("f")
        .long("foo")
        .help("Sets a custom foo")
        .takes_value(true))
    .arg(Arg::with_name("bar")
        .help("Sets baz")
        .required(true)
        .index(1))
    .arg(Arg::with_name("baz")
        .short("b")
        .multiple(true)
        .help("Sets the bazness"));

Then the user might have a config .foobarrc:

foo="some string"
bar=3

Your plan would work fine.

However, if they had this file:

fooo="some string"
bar=3

I can't see a good way to inform the user that option fooo is invalid; where do I get the list of valid names?

I could build it up at the same time as the clap::App, with some sort of function like

let valid_names = vec!();
let mut app = clap::App::new("foobar");

let new_opt = |s: &str| {
    valid_names.append(s);
    Arg::with_name("foo")
}

app = app
    .arg(new_opt("foo")
        .short("f")
        .long("foo")
        .help("Sets a custom foo")
        .takes_value(true))
    .arg(new_opt("bar")
        .help("Sets baz")
        .required(true)
        .index(1))
    .arg(new_opt("baz")
        .short("b")
        .multiple(true)
        .help("Sets the bazness"));

That's... actually not too bad; its still a bit awkward, though, and while it works for a single instance, it seems to me like it would be hard to extend to a library for use in many apps that handles configuration files as carefully and thoroughly as clap handles env::args().

If I wanted to write such a library, I can't really see how to do it effectively. I can't extend clap::App, to allow the user to do something like app.arg([...]).value_type(toml::Integer); I can't have the user pass my library a clap::App instance, because I can't extract any information about options from it (other than post-facto matches); the only way I can see to do it is to have thelibrary completely wrap clap, duplicating many of its functions, and producing its own, semi-internal clap::App instance, which seems like a much more painful path than if clap provided some sort of API for Arg::get_name(&self) -> &str, App::get_options(&self) -> Iter<Item=Arg>, or even better, a mutable form like you suggested.

Do you see what I mean? Right now, clap does a great job of handling many possibilities; however, it doesn't seem to leave many good ways for users to extend this functionality without forking clap or twisting their code in weird ways to work around clap's architecture.

What are your thoughts?

[WildCryptoFox]

Your suggestion of App::get_options(&self) -> Iter<Item=Arg> seems very suitable; also works very well for the macro builder, yaml parser and from_usage methods for App constructing. I recommend rename to get_args as you're returning the self.args.iter(). app.get_args().map(|arg| arg.get_name()).collect() for a usable Vec<&'app_arg str>.

As for value types - #146 already covers this issue.

[gilescope]

Having support for dot files out of the box would be wonderful (maybe as a 'feature'?). This could help standardise the way that rust cli apps are configured which seems like a good thing for the community. I know this is an old ticket, clap has come on so much in this time. Is clap able to do configs out of the box for 'free'?

[malarinv]

I ended up here looking for a rust port of https://github.com/bw2/ConfigArgParse
@wackywendell weren't you suggesting the same?

[dankamongmen]

@kbknapp you had asked for an example of such usage. I can think of two:

dialog: ncurses program for shell-launched terminal widgets
openvpn: VPN program

both allow all config file options to be stuffed in a file provided with --file (for dialog) or --config (for openvpn).

[CreepySkeleton]

@dankamongmen This issue has been merged into #748 since that one has more of a constructive discussion going. Perhaps you should go there.

[cmcqueen]

I agree with @malarinv — I ended up here essentially trying to find a Rust equivalent of ConfigArgParse package for Python. I've used ConfigArgParse in previous Python projects, and its feature set has been perfect for my needs in those projects.

In ConfigArgParse, I really like the is_config_file feature, where you can specify a path of a config file on the command line, and it will read config from that file (as well as being able to hard-code paths for config files).