Providing files from memory

[nikeee]

If I'm getting the docs right, it seems that in requestes referencing files, they would be read from disk (relative to context_dir I guess).

Is it possible to provide files to the hurl runner that are read from program memory instead from the disk? This would probably be useful when doing #233.

Maybe something like this?

enum FileSource {
    Disk { context_dir: str },
    Memory(FileProvider),
}

FileProvider could be a trait that provides methods to read a file, retrieving the content-length etc.
When it also provides writing abilities, it could be used to capture files from requests in-memory without writing them to disk, possibly adding a CLI option to retain captured files in memory.

Instead of the FileSource enum, a plain FileProvider could suffice, as one implementing IO access for the context_dir could be created on the fly in fn execute.

As I'm not that familiar with rust, there might also be an existing trait for this kind of virtual file system.

Looking at the source where the files are included, referencing in-memory data, as it seems that the parameters are passed as classic cURL CLI params:

hurl/packages/hurl/src/http/request_spec.rs

Lines 180 to 210 in e59e2ac

	for param in self.form.clone() {
	arguments.push("--data".to_string());
	arguments.push(format!("'{}'", param.curl_arg_escape()));
	}
	for param in self.multipart.clone() {
	arguments.push("-F".to_string());
	arguments.push(format!("'{}'", param.curl_arg(context_dir.clone())));
	}
	if !self.body.bytes().is_empty() {
	arguments.push("--data".to_string());
	match self.body.clone() {
	Body::Text(s) => {
	let prefix = if s.contains('\n') { "$" } else { "" };
	arguments.push(format!(
	"{}'{}'",
	prefix,
	s.replace("\\", "\\\\").replace("\n", "\\n")
	))
	}
	Body::Binary(bytes) => arguments.push(format!("$'{}'", encode_bytes(bytes))),
	Body::File(_, filename) => {
	let prefix = if context_dir.as_str() == "." {
	"".to_string()
	} else {
	format!("{}/", context_dir)
	};
	arguments.push(format!("'@{}{}'", prefix, filename))
	}
	}
	}

[fabricereix]

Hi @nikeee,
The code you show above in request_spec.rs is in fact not used for sending the HTTP request.
It is used to output the equivalent curl command-line in verbose mode.

test.hurl

GET http://localhost:8000/hello
Fruit: Raspberry
HTTP/1.0 200

$ hurl -v /tmp/test.hurl
* fail fast: true
* insecure: false
* follow redirect: false
* max redirect: 50
* ------------------------------------------------------------------------------
* executing entry 1
* 
* Cookie store:
* 
* Request
* GET http://localhost:8000/hello
* Fruit: Raspberry
* 
* request can be run with the following curl command:
* curl 'http://localhost:8000/hello' -H 'Fruit: Raspberry'
*
> GET /hello HTTP/1.1
> Host: localhost:8000
> Accept: */*
> Fruit: Raspberry
> User-Agent: hurl/1.5.0-snapshot
> 
< HTTP/1.0 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 12
< Server: Flask Server
< Date: Tue, 19 Oct 2021 12:10:51 GMT
< 
* 

[nikeee]

I implemented this in a fork, using the vfs crate.

You can see the diff here: master...nikeee:master

There is one failing test, though:

thread 'runner::body::tests::test_body_file_error' panicked at 'assertion failed: `(left == right)`
  left: `FileReadAccess { value: "/data.bin" }`,
 right: `FileReadAccess { value: "current_dir/data.bin" }`', packages/hurl/src/runner/body.rs:143:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This originates from current_dir being the root of the PhysicalFS, so it will be stripped on the output. This may be fixable somehow.

MemoryFS could be used instead of PhysicalFS in unit tests.

Is this something that would align with the project goals? If not, I'd maintain this in a separate fork.

[fabricereix]

Hi @nikeee,
We currently prefer to focus on the command-line usage for Hurl.
We may relook at your issue and your fork later on.
Thanks

[nikeee]

Since #284 is resolved and we'd be able to use Hurl as a library:
Is this something to re-consider?
Hurl really seems like the perfect tool to embed user-defined HTTP requests in an application.
For example, I've got a use case where a user can define how to upload a file. Depending on the target system, this may also involve multiple chained steps/requests.
The application could then invoke the hurl file with some parameters.
For performance reasons, we could just provide the file in-memory using some VFS implementation like mentioned above.

[jcamiel]

Hi @nikeee you can create a request body that uses memory by building an instance of HurlFile instead of parsing a Hurl file.
You have to look at packages/hurl_core/src/ast/core.rs: HurlFile, Request, Body struct. Basically, a request has a body field whose value is a Byte struct defined like this:

pub enum Bytes {
    Json(json::Value),
    Xml(String),
    MultilineString(MultilineString),
    OnelineString(Template),
    Base64(Base64),
    File(File),
    Hex(Hex),
}

For a memory file, you can use the Hex variant which can be initialized with a vec of bytes:

pub struct Hex {
    pub space0: Whitespace,
    pub value: Vec<u8>,
    pub encoded: String,
    pub space1: Whitespace,
}

Or use RawString variant if you have a text encoded memory buffer.
It's a bit cumbersome, but it can be done.

For the moment, we're not going to improve this use case.

The core focus of the project is:

• the file format,
• the CLI tool

The aim is to have an excellent CLI tool "à la" curl. The fact that one can use Hurl as a library is a very nice bonus, and we'll try to facilitate it. But our focus is the CLI and not Hurl as a crate. I hope you understand we're trying to stay focused on these features. In any case, I can assist you with sample code to how to forge a request by code, which can be done with the current version (1.8.0).

[nikeee]

I'll take a look at that, thank you for this approach. :)

The core focus of the project is

And that's great! Keep doing awesome stuff!