Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor alerts to make behavior clear #4019

Merged
merged 6 commits into from
Jun 5, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions docs/DEVELOPMENT-GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -411,21 +411,25 @@ struct s2n_stuffer alert_in;

'header_in' is a small 5-byte stuffer, which is used to read in a record header. Once that stuffer is full, and the size of the next record is determined (from that header), inward data is directed to the 'in' stuffer. The 'out' stuffer is for data that we are writing out; like an encrypted TLS record. 'alert_in' is for any TLS alert message that s2n-tls receives from its peer. s2n-tls treats all alerts as fatal, but we buffer the full alert message so that reason can be logged.

When past the handshake phase, s2n-tls supports full-duplex I/O. Separate threads or event handlers are free to call s2n_send and s2n_recv on the same connection. Because either a read or a write may cause a connection to be closed, there are two additional stuffers for storing outbound alert messages:
When past the handshake phase, s2n-tls supports full-duplex I/O. Separate threads or event handlers are free to call s2n_send and s2n_recv on the same connection. Because either a read or a write may cause a connection to be closed, there are two additional fields for storing outbound alert messages:

```c
struct s2n_stuffer reader_alert_out;
struct s2n_stuffer writer_alert_out;
uint8_t reader_alert_out;
uint8_t writer_alert_out;
```

this pattern means that both the reader thread and writer thread can create pending alert messages without needing any locks. If either the reader or writer generates an alert, it also sets the 'closing' state to 1.
This pattern means that both the reader thread and writer thread can create pending alert messages without needing any locks. If either the reader or writer generates an alert, it also sets the 'closed' states to 1.

```c
sig_atomic_t closing;
sig_atomic_t closed;
sig_atomic_t read_closed;
sig_atomic_t write_closed;
```

'closing' is an atomic, but even if it were not it can only be changed from 0 to 1, so an over-write is harmless. Every time a TLS record is fully-written, s2n_send() checks to see if closing is set to 1. If it is then the reader or writer alert message will be sent (writer takes priority, if both are present) and the connection will be closed. Once the closed is 1, no more I/O may be sent or received on the connection.
These fields are atomic. However because they are only ever changed from 0 to 1, an over-write would be harmless.

s2n-tls only sends fatal alerts during `s2n_shutdown()` or `s2n_shutdown_send()`.
Only one alert is sent, so writer alerts take priority if both are present.
If no alerts are present, then a generic close_notify will be sent instead.

## s2n-tls and entropy

Expand Down
2 changes: 0 additions & 2 deletions tests/cbmc/sources/make_common_datastructures.c
Original file line number Diff line number Diff line change
Expand Up @@ -784,8 +784,6 @@ void cbmc_populate_s2n_connection(struct s2n_connection *s2n_connection)
cbmc_populate_s2n_stuffer(&(s2n_connection->in));
cbmc_populate_s2n_stuffer(&(s2n_connection->out));
cbmc_populate_s2n_stuffer(&(s2n_connection->alert_in));
cbmc_populate_s2n_stuffer(&(s2n_connection->reader_alert_out));
cbmc_populate_s2n_stuffer(&(s2n_connection->writer_alert_out));
cbmc_populate_s2n_handshake(&(s2n_connection->handshake));
cbmc_populate_s2n_blob(&(s2n_connection->status_response));
cbmc_populate_s2n_blob(&(s2n_connection->ct_response));
Expand Down
5 changes: 2 additions & 3 deletions tests/unit/s2n_alerts_protocol_test.c
Original file line number Diff line number Diff line change
Expand Up @@ -294,10 +294,9 @@ int main(int argc, char **argv)
/* Remove any blinding so that we can immediately call shutdown */
failed_conn->delay = 0;

/* In most cases, the alert will not be sent until we attempt to shutdown */
/* The alert will not be sent until we attempt to shutdown */
EXPECT_SUCCESS(s2n_shutdown_send(failed_conn, &blocked));
EXPECT_EQUAL(expected_alert == S2N_TLS_ALERT_CLOSE_NOTIFY,
failed_conn->close_notify_queued);
EXPECT_TRUE(failed_conn->alert_sent);

/**
*= https://tools.ietf.org/rfc/rfc8446#section-6.2
Expand Down
182 changes: 89 additions & 93 deletions tests/unit/s2n_alerts_test.c
Original file line number Diff line number Diff line change
Expand Up @@ -274,110 +274,106 @@ int main(int argc, char **argv)
}
}

/* Test s2n_queue_writer_close_alert_warning */
/* Test s2n_queue_reader_alert */
{
/* Safety check */
EXPECT_FAILURE_WITH_ERRNO(s2n_queue_writer_close_alert_warning(NULL), S2N_ERR_NULL);

/* Does not send alert if alerts not supported */
lrstewart marked this conversation as resolved.
Show resolved Hide resolved
if (s2n_is_tls13_fully_supported()) {
struct s2n_config *config;
EXPECT_NOT_NULL(config = s2n_config_new());

struct s2n_connection *conn;
EXPECT_NOT_NULL(conn = s2n_connection_new(S2N_CLIENT));
EXPECT_SUCCESS(s2n_connection_set_config(conn, config));

EXPECT_EQUAL(s2n_stuffer_data_available(&conn->writer_alert_out), 0);

/* Writes alert by default */
EXPECT_SUCCESS(s2n_queue_writer_close_alert_warning(conn));
EXPECT_EQUAL(s2n_stuffer_data_available(&conn->writer_alert_out), ALERT_LEN);

/* Wipe error */
EXPECT_SUCCESS(s2n_stuffer_wipe(&conn->writer_alert_out));

/* Does not write alert when alerts not supported (when QUIC mode enabled) */
EXPECT_SUCCESS(s2n_config_enable_quic(config));
EXPECT_SUCCESS(s2n_queue_writer_close_alert_warning(conn));
EXPECT_EQUAL(s2n_stuffer_data_available(&conn->writer_alert_out), 0);

EXPECT_SUCCESS(s2n_connection_free(conn));
EXPECT_SUCCESS(s2n_config_free(config));
}
}

/* Test s2n_queue_reader_alert
* Since s2n_queue_reader_alert is static, we'll test it indirectly via s2n_queue_reader_handshake_failure_alert */
{
/* Safety check */
EXPECT_FAILURE_WITH_ERRNO(s2n_queue_reader_handshake_failure_alert(NULL), S2N_ERR_NULL);
/* Safety */
EXPECT_FAILURE_WITH_ERRNO(s2n_queue_reader_unsupported_protocol_version_alert(NULL),
S2N_ERR_NULL);
EXPECT_FAILURE_WITH_ERRNO(s2n_queue_reader_handshake_failure_alert(NULL),
S2N_ERR_NULL);

/* Does not send alert if alerts not supported */
if (s2n_is_tls13_fully_supported()) {
struct s2n_config *config;
EXPECT_NOT_NULL(config = s2n_config_new());

struct s2n_connection *conn;
EXPECT_NOT_NULL(conn = s2n_connection_new(S2N_CLIENT));
EXPECT_SUCCESS(s2n_connection_set_config(conn, config));

EXPECT_EQUAL(s2n_stuffer_data_available(&conn->reader_alert_out), 0);

/* Writes alert by default */
EXPECT_SUCCESS(s2n_queue_reader_handshake_failure_alert(conn));
EXPECT_EQUAL(s2n_stuffer_data_available(&conn->reader_alert_out), ALERT_LEN);
DEFER_CLEANUP(struct s2n_connection *conn = s2n_connection_new(S2N_CLIENT),
s2n_connection_ptr_free);
EXPECT_NOT_NULL(conn);

/* Wipe error */
EXPECT_SUCCESS(s2n_stuffer_wipe(&conn->reader_alert_out));
/* Alert queued */
EXPECT_SUCCESS(s2n_queue_reader_unsupported_protocol_version_alert(conn));
EXPECT_EQUAL(conn->reader_alert_out, S2N_TLS_ALERT_PROTOCOL_VERSION);

/* Does not write alert when alerts not supported (when QUIC mode enabled) */
EXPECT_SUCCESS(s2n_config_enable_quic(config));
EXPECT_SUCCESS(s2n_queue_reader_handshake_failure_alert(conn));
EXPECT_EQUAL(s2n_stuffer_data_available(&conn->reader_alert_out), 0);
/* New alert not queued if alert already set */
EXPECT_SUCCESS(s2n_queue_reader_handshake_failure_alert(conn));
EXPECT_EQUAL(conn->reader_alert_out, S2N_TLS_ALERT_PROTOCOL_VERSION);

EXPECT_SUCCESS(s2n_connection_free(conn));
EXPECT_SUCCESS(s2n_config_free(config));
}
}
/* New alert queued if old alert cleared */
conn->reader_alert_out = 0;
EXPECT_SUCCESS(s2n_queue_reader_handshake_failure_alert(conn));
EXPECT_EQUAL(conn->reader_alert_out, S2N_TLS_ALERT_HANDSHAKE_FAILURE);
};

/* Test s2n_alerts_close_if_fatal */
/* Test s2n_alerts_write_error_or_close_notify */
{
DEFER_CLEANUP(struct s2n_connection *conn = s2n_connection_new(S2N_CLIENT),
s2n_connection_ptr_free);
EXPECT_NOT_NULL(conn);
EXPECT_OK(s2n_connection_set_secrets(conn));
const uint8_t expected_alert = S2N_TLS_ALERT_INTERNAL_ERROR;
const uint8_t wrong_alert = S2N_TLS_ALERT_CERTIFICATE_UNKNOWN;

DEFER_CLEANUP(struct s2n_test_io_pair io_pair = { 0 }, s2n_io_pair_close);
EXPECT_SUCCESS(s2n_io_pair_init_non_blocking(&io_pair));
EXPECT_SUCCESS(s2n_connection_set_io_pair(conn, &io_pair));
/* Test: if no alerts set, close_notify sent */
{
DEFER_CLEANUP(struct s2n_connection *conn = s2n_connection_new(S2N_CLIENT),
s2n_connection_ptr_free);
EXPECT_NOT_NULL(conn);

EXPECT_OK(s2n_alerts_write_error_or_close_notify(conn));

/* Verify record written */
uint8_t level = 0, code = 0;
EXPECT_SUCCESS(s2n_stuffer_skip_read(&conn->out, S2N_TLS_RECORD_HEADER_LENGTH));
EXPECT_SUCCESS(s2n_stuffer_read_uint8(&conn->out, &level));
EXPECT_EQUAL(level, S2N_TLS_ALERT_LEVEL_WARNING);
EXPECT_SUCCESS(s2n_stuffer_read_uint8(&conn->out, &code));
EXPECT_EQUAL(code, S2N_TLS_ALERT_CLOSE_NOTIFY);
EXPECT_EQUAL(s2n_stuffer_data_available(&conn->out), 0);
};

s2n_blocked_status blocked = S2N_NOT_BLOCKED;
/* Test: if only reader alert set, reader alert sent */
{
DEFER_CLEANUP(struct s2n_connection *conn = s2n_connection_new(S2N_CLIENT),
s2n_connection_ptr_free);
EXPECT_NOT_NULL(conn);
conn->reader_alert_out = expected_alert;

EXPECT_OK(s2n_alerts_write_error_or_close_notify(conn));

/* Verify record written */
uint8_t level = 0, code = 0;
EXPECT_SUCCESS(s2n_stuffer_skip_read(&conn->out, S2N_TLS_RECORD_HEADER_LENGTH));
EXPECT_SUCCESS(s2n_stuffer_read_uint8(&conn->out, &level));
EXPECT_EQUAL(level, S2N_TLS_ALERT_LEVEL_FATAL);
EXPECT_SUCCESS(s2n_stuffer_read_uint8(&conn->out, &code));
EXPECT_EQUAL(code, expected_alert);
EXPECT_EQUAL(s2n_stuffer_data_available(&conn->out), 0);
};

/* Sanity check: s2n_flush doesn't close the connection */
conn->write_closing = false;
EXPECT_SUCCESS(s2n_flush(conn, &blocked));
EXPECT_FALSE(conn->write_closing);
/* Test: if both alerts set, writer alert sent */
{
DEFER_CLEANUP(struct s2n_connection *conn = s2n_connection_new(S2N_CLIENT),
s2n_connection_ptr_free);
EXPECT_NOT_NULL(conn);
conn->writer_alert_out = expected_alert;
conn->reader_alert_out = wrong_alert;

EXPECT_OK(s2n_alerts_write_error_or_close_notify(conn));

/* Verify record written */
uint8_t level = 0, code = 0;
EXPECT_SUCCESS(s2n_stuffer_skip_read(&conn->out, S2N_TLS_RECORD_HEADER_LENGTH));
EXPECT_SUCCESS(s2n_stuffer_read_uint8(&conn->out, &level));
EXPECT_EQUAL(level, S2N_TLS_ALERT_LEVEL_FATAL);
EXPECT_SUCCESS(s2n_stuffer_read_uint8(&conn->out, &code));
EXPECT_EQUAL(code, expected_alert);
EXPECT_EQUAL(s2n_stuffer_data_available(&conn->out), 0);
};

/* Test: a fatal reader alert closes the connection */
conn->write_closing = false;
EXPECT_SUCCESS(s2n_queue_reader_handshake_failure_alert(conn));
EXPECT_SUCCESS(s2n_flush(conn, &blocked));
EXPECT_TRUE(conn->write_closing);

/* Test: a close_notify alert closes the connection
* This is our only writer alert, and technically it's a warning.
*/
conn->write_closing = false;
EXPECT_SUCCESS(s2n_queue_writer_close_alert_warning(conn));
EXPECT_SUCCESS(s2n_flush(conn, &blocked));
EXPECT_TRUE(conn->write_closing);

/* Test: a no_renegotiation alert does not close the connection */
conn->write_closing = false;
EXPECT_OK(s2n_queue_reader_no_renegotiation_alert(conn));
EXPECT_SUCCESS(s2n_flush(conn, &blocked));
EXPECT_FALSE(conn->write_closing);
/* If alerts not supported, no alerts sent */
{
DEFER_CLEANUP(struct s2n_connection *conn = s2n_connection_new(S2N_CLIENT),
s2n_connection_ptr_free);
EXPECT_NOT_NULL(conn);
conn->quic_enabled = true;
conn->writer_alert_out = expected_alert;
conn->reader_alert_out = wrong_alert;

EXPECT_OK(s2n_alerts_write_error_or_close_notify(conn));
EXPECT_EQUAL(s2n_stuffer_data_available(&conn->out), 0);
};
}

END_TEST();
Expand Down
4 changes: 2 additions & 2 deletions tests/unit/s2n_client_extensions_test.c
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,7 @@ int main(int argc, char **argv)
EXPECT_BYTEARRAY_EQUAL(received_server_name, sent_server_name, strlen(received_server_name));

EXPECT_SUCCESS(s2n_shutdown(server_conn, &server_blocked));
EXPECT_EQUAL(server_conn->close_notify_queued, 1);
EXPECT_TRUE(server_conn->alert_sent);

EXPECT_SUCCESS(s2n_connection_free(server_conn));

Expand Down Expand Up @@ -621,7 +621,7 @@ int main(int argc, char **argv)
EXPECT_EQUAL(server_conn->secure_renegotiation, 1);

EXPECT_SUCCESS(s2n_shutdown(server_conn, &server_blocked));
EXPECT_EQUAL(server_conn->close_notify_queued, 1);
EXPECT_TRUE(server_conn->alert_sent);

EXPECT_SUCCESS(s2n_connection_free(server_conn));

Expand Down
21 changes: 16 additions & 5 deletions tests/unit/s2n_client_hello_request_test.c
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,6 @@ int main(int argc, char **argv)
EXPECT_OK(s2n_test_send_and_recv(server_conn, client_conn));
EXPECT_OK(s2n_test_send_and_recv(client_conn, server_conn));
EXPECT_TRUE(s2n_connection_check_io_status(client_conn, S2N_IO_FULL_DUPLEX));
EXPECT_FALSE(client_conn->write_closing);
}
};

Expand Down Expand Up @@ -513,6 +512,7 @@ int main(int argc, char **argv)
EXPECT_SUCCESS(s2n_connection_set_config(client_conn, config_with_reneg_cb));
EXPECT_SUCCESS(s2n_connection_set_cipher_preferences(client_conn, "test_all"));
EXPECT_SUCCESS(s2n_config_set_renegotiate_request_cb(config_with_reneg_cb, s2n_test_reneg_req_cb, &ctx));
EXPECT_SUCCESS(s2n_connection_set_blinding(client_conn, S2N_SELF_SERVICE_BLINDING));

DEFER_CLEANUP(struct s2n_connection *server_conn = s2n_connection_new(S2N_SERVER),
s2n_connection_ptr_free);
Expand All @@ -524,6 +524,9 @@ int main(int argc, char **argv)
EXPECT_SUCCESS(s2n_io_pair_init_non_blocking(&io_pair));
EXPECT_SUCCESS(s2n_connections_set_io_pair(client_conn, server_conn, &io_pair));

uint8_t buffer[1] = { 0 };
s2n_blocked_status blocked = S2N_NOT_BLOCKED;

/* Force an SSLv3 handshake */
client_conn->client_protocol_version = S2N_SSLv3;
client_conn->actual_protocol_version = S2N_SSLv3;
Expand All @@ -534,12 +537,20 @@ int main(int argc, char **argv)
/* Send the hello request message. */
EXPECT_OK(s2n_send_client_hello_request(server_conn));

/* handshake_failure alert sent and received */
EXPECT_OK(s2n_test_send_and_recv(server_conn, client_conn));
EXPECT_ERROR_WITH_ERRNO(s2n_test_send_and_recv(client_conn, server_conn), S2N_ERR_ALERT);
EXPECT_EQUAL(s2n_connection_get_alert(server_conn), S2N_TLS_ALERT_HANDSHAKE_FAILURE);
/* handshake_failure alert queued */
EXPECT_FAILURE_WITH_ERRNO(s2n_recv(client_conn, buffer, sizeof(buffer), &blocked), S2N_ERR_BAD_MESSAGE);
EXPECT_TRUE(s2n_connection_check_io_status(client_conn, S2N_IO_CLOSED));

/* handshake_failure alert send.
* Skip blinding. */
EXPECT_TRUE(client_conn->delay > 0);
client_conn->delay = 0;
EXPECT_SUCCESS(s2n_shutdown_send(client_conn, &blocked));

/* handshake_failure alert received */
EXPECT_FAILURE_WITH_ERRNO(s2n_recv(server_conn, buffer, sizeof(buffer), &blocked), S2N_ERR_ALERT);
EXPECT_TRUE(s2n_connection_check_io_status(server_conn, S2N_IO_CLOSED));
EXPECT_EQUAL(s2n_connection_get_alert(server_conn), S2N_TLS_ALERT_HANDSHAKE_FAILURE);

/* Callback triggered */
EXPECT_NOT_NULL(client_conn->config->renegotiate_request_cb);
Expand Down
17 changes: 5 additions & 12 deletions tests/unit/s2n_client_supported_versions_extension_test.c
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,6 @@
#define PROTOCOL_VERSION_ALERT 70
#define GREASED_SUPPORTED_VERSION_EXTENSION_VALUES 0x0A0A, 0x1A1A, 0x2A2A, 0x3A3A, 0x4A4A, 0x5A5A, 0x6A6A, 0x7A7A, 0x8A8A, 0x9A9A, 0xAAAA, 0xBABA, 0xCACA, 0xDADA, 0xEAEA, 0xFAFA

int get_alert(struct s2n_connection *conn)
{
uint8_t error[2];
POSIX_GUARD(s2n_stuffer_read_bytes(&conn->reader_alert_out, error, 2));
return error[1];
}

int write_test_supported_versions_list(struct s2n_stuffer *list, uint8_t *supported_versions, uint8_t length)
{
POSIX_GUARD(s2n_stuffer_write_uint8(list, length * S2N_TLS_PROTOCOL_VERSION_LEN));
Expand Down Expand Up @@ -296,7 +289,7 @@ int main(int argc, char **argv)

EXPECT_FAILURE_WITH_ERRNO(s2n_client_supported_versions_extension.recv(server_conn, &extension),
S2N_ERR_PROTOCOL_VERSION_UNSUPPORTED);
EXPECT_EQUAL(get_alert(server_conn), PROTOCOL_VERSION_ALERT);
EXPECT_EQUAL(server_conn->reader_alert_out, PROTOCOL_VERSION_ALERT);

EXPECT_SUCCESS(s2n_connection_free(server_conn));
EXPECT_SUCCESS(s2n_stuffer_free(&extension));
Expand All @@ -315,7 +308,7 @@ int main(int argc, char **argv)

EXPECT_FAILURE_WITH_ERRNO(s2n_client_supported_versions_extension.recv(server_conn, &extension),
S2N_ERR_UNKNOWN_PROTOCOL_VERSION);
EXPECT_EQUAL(get_alert(server_conn), PROTOCOL_VERSION_ALERT);
EXPECT_EQUAL(server_conn->reader_alert_out, PROTOCOL_VERSION_ALERT);

EXPECT_SUCCESS(s2n_connection_free(server_conn));
EXPECT_SUCCESS(s2n_stuffer_free(&extension));
Expand All @@ -333,7 +326,7 @@ int main(int argc, char **argv)
EXPECT_SUCCESS(s2n_stuffer_write_uint8(&extension, 13));

EXPECT_FAILURE_WITH_ERRNO(s2n_client_supported_versions_extension.recv(server_conn, &extension), S2N_ERR_BAD_MESSAGE);
EXPECT_EQUAL(get_alert(server_conn), PROTOCOL_VERSION_ALERT);
EXPECT_EQUAL(server_conn->reader_alert_out, PROTOCOL_VERSION_ALERT);

EXPECT_SUCCESS(s2n_connection_free(server_conn));
EXPECT_SUCCESS(s2n_stuffer_free(&extension));
Expand All @@ -353,7 +346,7 @@ int main(int argc, char **argv)
EXPECT_SUCCESS(s2n_stuffer_write_uint16(&extension, 0x0303));

EXPECT_FAILURE_WITH_ERRNO(s2n_client_supported_versions_extension.recv(server_conn, &extension), S2N_ERR_BAD_MESSAGE);
EXPECT_EQUAL(get_alert(server_conn), PROTOCOL_VERSION_ALERT);
EXPECT_EQUAL(server_conn->reader_alert_out, PROTOCOL_VERSION_ALERT);

EXPECT_SUCCESS(s2n_connection_free(server_conn));
EXPECT_SUCCESS(s2n_stuffer_free(&extension));
Expand All @@ -373,7 +366,7 @@ int main(int argc, char **argv)
EXPECT_SUCCESS(s2n_stuffer_write_uint8(&extension, 0x03));

EXPECT_FAILURE_WITH_ERRNO(s2n_client_supported_versions_extension.recv(server_conn, &extension), S2N_ERR_BAD_MESSAGE);
EXPECT_EQUAL(get_alert(server_conn), PROTOCOL_VERSION_ALERT);
EXPECT_EQUAL(server_conn->reader_alert_out, PROTOCOL_VERSION_ALERT);

EXPECT_SUCCESS(s2n_connection_free(server_conn));
EXPECT_SUCCESS(s2n_stuffer_free(&extension));
Expand Down
5 changes: 1 addition & 4 deletions tests/unit/s2n_record_write_test.c
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,8 @@ int main(int argc, char *argv[])
EXPECT_SUCCESS(s2n_stuffer_growable_alloc(&out, 0));
EXPECT_SUCCESS(s2n_connection_set_send_io_stuffer(&out, server_conn));

EXPECT_SUCCESS(s2n_queue_writer_close_alert_warning(server_conn));

s2n_blocked_status blocked = S2N_NOT_BLOCKED;
EXPECT_ERROR_WITH_ERRNO(s2n_negotiate_until_message(server_conn, &blocked, SERVER_HELLO),
S2N_ERR_CLOSED);
EXPECT_SUCCESS(s2n_shutdown_send(server_conn, &blocked));

uint8_t content_type = 0;
EXPECT_SUCCESS(s2n_stuffer_read_uint8(&out, &content_type));
Expand Down
Loading