Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement type checking/validation for filters #4364

Merged
merged 46 commits into from
Apr 9, 2023
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
ae6bcdf
Initial implementation of filter type validation
dnsge Nov 9, 2022
eb11e76
Extract expression classes to own files
dnsge Feb 6, 2023
4ac4df1
R-value for ListExpression constructor
dnsge Nov 15, 2022
22540a5
Add type validation error log to UnaryOperation
dnsge Nov 15, 2022
bcc4a58
Implement proper type synthesizer
dnsge Feb 8, 2023
dcd608c
Move TokenType to Tokenizer
dnsge Feb 8, 2023
6fa70b0
Rename filterparser namespace to chatterino::filters
dnsge Feb 8, 2023
bd170c1
Rename filters/parser/* to filters/lang/*
dnsge Feb 10, 2023
1a0b3ff
Separate parsing and usage of filters
dnsge Feb 10, 2023
d5f9d9a
Add tests for filters
dnsge Feb 10, 2023
f4ac0b4
Remove old comments
dnsge Feb 11, 2023
63f71bd
Improve filter debug output
dnsge Feb 11, 2023
7f23b9c
Fix startswith endswith string list bug
dnsge Feb 11, 2023
847c782
Update author.badges type to StringList
dnsge Feb 11, 2023
0c0cefd
Add extensive evaluation semantics tests
dnsge Feb 11, 2023
c83abc3
Improve filter error reporting
dnsge Feb 11, 2023
5fef124
Silghtly adjust how ChannelFilterEditorDialog saves filters
dnsge Feb 11, 2023
a06ffb8
Reword some error messages
dnsge Feb 11, 2023
63a611a
Clang-tidy suggestions
dnsge Feb 13, 2023
975327b
Update CHANGELOG.md
dnsge Feb 13, 2023
85a15e2
Add separate typing context map
dnsge Feb 18, 2023
246a8e5
Merge branch 'master' into feat/filter-type-checking
dnsge Feb 21, 2023
7e90ae7
Fix changelog entry position
dnsge Feb 21, 2023
6cf05e5
Merge branch 'master' into feat/filter-type-checking
dnsge Mar 30, 2023
0e9cfba
Merge branch 'master' into feat/filter-type-checking
pajlada Apr 7, 2023
e26c397
dev: rename and document MESSAGE_TYPING_CONTEXT
dnsge Apr 7, 2023
5474c93
dev: implement PossibleType with std::variant
dnsge Apr 7, 2023
ddb1cc8
dev: add warning to IllTyped::expr
dnsge Apr 7, 2023
8acf85e
fix: matchless regex group matching returns empty string
dnsge Apr 8, 2023
b294db3
Merge branch 'master' into feat/filter-type-checking
dnsge Apr 8, 2023
afecefd
dev: replace PossibleType wrapper with std::variant
dnsge Apr 8, 2023
ac97b89
dev: explicilty implement operator!=
dnsge Apr 8, 2023
613c4bf
constify FilterRecord members
pajlada Apr 9, 2023
32355ff
Filter.cpp: absolute include path
pajlada Apr 9, 2023
0ef4f4c
FilterParser.cpp: absolute include path
pajlada Apr 9, 2023
1f612b8
Types.cpp: absolute include path
pajlada Apr 9, 2023
f8056bb
Types isList: fix parameter name
pajlada Apr 9, 2023
a0ae6df
binary operation absolute include path
pajlada Apr 9, 2023
2c84e9b
expression absolute include path
pajlada Apr 9, 2023
062cb97
listexpression absolute include path
pajlada Apr 9, 2023
c6b4279
regex expression: absolute include path
pajlada Apr 9, 2023
d8a67a5
unary operation absolute include path
pajlada Apr 9, 2023
3039311
value expression absolute include path
pajlada Apr 9, 2023
3279690
channe lfilter editor dialog absolute include path
pajlada Apr 9, 2023
b24c345
filter record absolute include path
pajlada Apr 9, 2023
9752c21
Merge remote-tracking branch 'origin/master' into feat/filter-type-ch…
pajlada Apr 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
- Minor: Added support for HTTP and Socks5 proxies through environment variables. (#4321)
- Minor: Added crashpad to capture crashes on Windows locally. See PR for build/crash analysis instructions. (#4351)
- Minor: Github releases now include flatpakref files for nightly builds
- Minor: Added better filter validation and error messages. (#4364)
- Bugfix: Fixed User Card moderation actions not working after Twitch IRC chat command deprecation. (#4378)
- Bugfix: Fixed User Card broadcaster actions (mod, unmod, vip, unvip) not working after Twitch IRC chat command deprecation. (#4387)
- Bugfix: Fixed crash that would occur when performing certain actions after removing all tabs. (#4271)
Expand Down
26 changes: 20 additions & 6 deletions src/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -74,12 +74,26 @@ set(SOURCE_FILES
controllers/filters/FilterRecord.hpp
controllers/filters/FilterSet.cpp
controllers/filters/FilterSet.hpp
controllers/filters/parser/FilterParser.cpp
controllers/filters/parser/FilterParser.hpp
controllers/filters/parser/Tokenizer.cpp
controllers/filters/parser/Tokenizer.hpp
controllers/filters/parser/Types.cpp
controllers/filters/parser/Types.hpp
controllers/filters/lang/expressions/Expression.cpp
controllers/filters/lang/expressions/Expression.hpp
controllers/filters/lang/expressions/BinaryOperation.cpp
controllers/filters/lang/expressions/BinaryOperation.hpp
controllers/filters/lang/expressions/ListExpression.cpp
controllers/filters/lang/expressions/ListExpression.hpp
controllers/filters/lang/expressions/RegexExpression.cpp
controllers/filters/lang/expressions/RegexExpression.hpp
controllers/filters/lang/expressions/UnaryOperation.hpp
controllers/filters/lang/expressions/UnaryOperation.cpp
controllers/filters/lang/expressions/ValueExpression.cpp
controllers/filters/lang/expressions/ValueExpression.hpp
controllers/filters/lang/Filter.cpp
controllers/filters/lang/Filter.hpp
controllers/filters/lang/FilterParser.cpp
controllers/filters/lang/FilterParser.hpp
controllers/filters/lang/Tokenizer.cpp
controllers/filters/lang/Tokenizer.hpp
controllers/filters/lang/Types.cpp
controllers/filters/lang/Types.hpp

controllers/highlights/BadgeHighlightModel.cpp
controllers/highlights/BadgeHighlightModel.hpp
Expand Down
38 changes: 27 additions & 11 deletions src/controllers/filters/FilterRecord.cpp
Original file line number Diff line number Diff line change
@@ -1,22 +1,37 @@
#include "controllers/filters/FilterRecord.hpp"
#include "FilterRecord.hpp"

#include "controllers/filters/lang/Filter.hpp"

namespace chatterino {

FilterRecord::FilterRecord(const QString &name, const QString &filter)
: name_(name)
, filter_(filter)
, id_(QUuid::createUuid())
, parser_(std::make_unique<filterparser::FilterParser>(filter))
: FilterRecord(name, filter, QUuid::createUuid())
{
}

FilterRecord::FilterRecord(const QString &name, const QString &filter,
const QUuid &id)
: name_(name)
, filter_(filter)
, filterText_(filter)
, id_(id)
, parser_(std::make_unique<filterparser::FilterParser>(filter))
{
using namespace filters;
auto result = Filter::fromString(filter);
if (std::holds_alternative<Filter>(result))
{
this->filter_ =
std::make_unique<Filter>(std::move(std::get<Filter>(result)));

if (this->filter_->returnType() != Type::Bool)
{
// Only accept Bool results
this->filter_ = nullptr;
}
}
else
{
this->filter_ = nullptr;
}
}

const QString &FilterRecord::getName() const
Expand All @@ -26,7 +41,7 @@ const QString &FilterRecord::getName() const

const QString &FilterRecord::getFilter() const
{
return this->filter_;
return this->filterText_;
}

const QUuid &FilterRecord::getId() const
Expand All @@ -36,12 +51,13 @@ const QUuid &FilterRecord::getId() const

bool FilterRecord::valid() const
{
return this->parser_->valid();
return this->filter_ != nullptr;
}

bool FilterRecord::filter(const filterparser::ContextMap &context) const
bool FilterRecord::filter(const filters::ContextMap &context) const
dnsge marked this conversation as resolved.
Show resolved Hide resolved
dnsge marked this conversation as resolved.
Show resolved Hide resolved
{
return this->parser_->execute(context);
assert(this->valid());
return this->filter_->execute(context).toBool();
}

bool FilterRecord::operator==(const FilterRecord &other) const
Expand Down
8 changes: 4 additions & 4 deletions src/controllers/filters/FilterRecord.hpp
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#pragma once

#include "controllers/filters/parser/FilterParser.hpp"
#include "controllers/filters/lang/Filter.hpp"
#include "util/RapidjsonHelpers.hpp"
#include "util/RapidJsonSerializeQString.hpp"

Expand Down Expand Up @@ -28,16 +28,16 @@ class FilterRecord

bool valid() const;

bool filter(const filterparser::ContextMap &context) const;
bool filter(const filters::ContextMap &context) const;

bool operator==(const FilterRecord &other) const;

private:
QString name_;
QString filter_;
QString filterText_;
QUuid id_;

std::unique_ptr<filterparser::FilterParser> parser_;
std::unique_ptr<filters::Filter> filter_;
};

using FilterRecordPtr = std::shared_ptr<FilterRecord>;
Expand Down
3 changes: 1 addition & 2 deletions src/controllers/filters/FilterSet.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,7 @@ bool FilterSet::filter(const MessagePtr &m, ChannelPtr channel) const
if (this->filters_.size() == 0)
return true;

filterparser::ContextMap context =
filterparser::buildContextMap(m, channel.get());
filters::ContextMap context = filters::buildContextMap(m, channel.get());
dnsge marked this conversation as resolved.
Show resolved Hide resolved
for (const auto &f : this->filters_.values())
{
if (!f->valid() || !f->filter(context))
Expand Down
155 changes: 155 additions & 0 deletions src/controllers/filters/lang/Filter.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
#include "Filter.hpp"

#include "Application.hpp"
#include "common/Channel.hpp"
#include "controllers/filters/lang/FilterParser.hpp"
#include "messages/Message.hpp"
#include "providers/twitch/TwitchBadge.hpp"
#include "providers/twitch/TwitchChannel.hpp"
#include "providers/twitch/TwitchIrcServer.hpp"

namespace chatterino::filters {

ContextMap buildContextMap(const MessagePtr &m, chatterino::Channel *channel)
{
auto watchingChannel = chatterino::getApp()->twitch->watchingChannel.get();

/* Known Identifiers
*
* author.badges
* author.color
* author.name
* author.no_color
* author.subbed
* author.sub_length
*
* channel.name
* channel.watching
*
* flags.highlighted
* flags.points_redeemed
* flags.sub_message
* flags.system_message
* flags.reward_message
* flags.first_message
* flags.elevated_message
* flags.cheer_message
* flags.whisper
* flags.reply
* flags.automod
*
* message.content
* message.length
*
*/

using MessageFlag = chatterino::MessageFlag;

QStringList badges;
dnsge marked this conversation as resolved.
Show resolved Hide resolved
badges.reserve(m->badges.size());
for (const auto &e : m->badges)
{
badges << e.key_;
}

bool watching = !watchingChannel->getName().isEmpty() &&
dnsge marked this conversation as resolved.
Show resolved Hide resolved
watchingChannel->getName().compare(
m->channelName, Qt::CaseInsensitive) == 0;

bool subscribed = false;
int subLength = 0;
for (const auto &subBadge : {"subscriber", "founder"})
{
if (!badges.contains(subBadge))
{
continue;
}
subscribed = true;
if (m->badgeInfos.find(subBadge) != m->badgeInfos.end())
{
subLength = m->badgeInfos.at(subBadge).toInt();
}
}
ContextMap vars = {
dnsge marked this conversation as resolved.
Show resolved Hide resolved
{"author.badges", std::move(badges)},
{"author.color", m->usernameColor},
{"author.name", m->displayName},
{"author.no_color", !m->usernameColor.isValid()},
{"author.subbed", subscribed},
{"author.sub_length", subLength},

{"channel.name", m->channelName},
{"channel.watching", watching},

{"flags.highlighted", m->flags.has(MessageFlag::Highlighted)},
{"flags.points_redeemed", m->flags.has(MessageFlag::RedeemedHighlight)},
{"flags.sub_message", m->flags.has(MessageFlag::Subscription)},
{"flags.system_message", m->flags.has(MessageFlag::System)},
{"flags.reward_message",
m->flags.has(MessageFlag::RedeemedChannelPointReward)},
{"flags.first_message", m->flags.has(MessageFlag::FirstMessage)},
{"flags.elevated_message", m->flags.has(MessageFlag::ElevatedMessage)},
{"flags.cheer_message", m->flags.has(MessageFlag::CheerMessage)},
{"flags.whisper", m->flags.has(MessageFlag::Whisper)},
{"flags.reply", m->flags.has(MessageFlag::ReplyMessage)},
{"flags.automod", m->flags.has(MessageFlag::AutoMod)},

{"message.content", m->messageText},
{"message.length", m->messageText.length()},
};
{
using namespace chatterino;
auto *tc = dynamic_cast<TwitchChannel *>(channel);
if (channel && !channel->isEmpty() && tc)
dnsge marked this conversation as resolved.
Show resolved Hide resolved
dnsge marked this conversation as resolved.
Show resolved Hide resolved
{
vars["channel.live"] = tc->isLive();
}
else
{
vars["channel.live"] = false;
}
}
return vars;
}

FilterResult Filter::fromString(const QString &str)
dnsge marked this conversation as resolved.
Show resolved Hide resolved
dnsge marked this conversation as resolved.
Show resolved Hide resolved
{
FilterParser parser(str);

if (parser.valid())
{
auto exp = parser.release();
auto typ = parser.returnType();
return Filter(std::move(exp), typ);
}

return FilterError{parser.errors().join("\n")};
}

Filter::Filter(ExpressionPtr expression, Type returnType)
dnsge marked this conversation as resolved.
Show resolved Hide resolved
: expression_(std::move(expression))
, returnType_(returnType)
{
}

Type Filter::returnType() const
{
return this->returnType_;
}

QVariant Filter::execute(const ContextMap &context) const
{
return this->expression_->execute(context);
}

QString Filter::filterString() const
{
return this->expression_->filterString();
}

QString Filter::debugString() const
{
return this->expression_->debug();
}

} // namespace chatterino::filters
48 changes: 48 additions & 0 deletions src/controllers/filters/lang/Filter.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#pragma once

#include "controllers/filters/lang/expressions/Expression.hpp"
#include "controllers/filters/lang/Types.hpp"

#include <QString>

#include <memory>
#include <variant>

namespace chatterino {

class Channel;
struct Message;
using MessagePtr = std::shared_ptr<const Message>;

} // namespace chatterino

namespace chatterino::filters {

ContextMap buildContextMap(const MessagePtr &m, chatterino::Channel *channel);

class Filter;
struct FilterError {
QString message;
};

using FilterResult = std::variant<Filter, FilterError>;
dnsge marked this conversation as resolved.
Show resolved Hide resolved

class Filter
{
public:
static FilterResult fromString(const QString &str);

Type returnType() const;
QVariant execute(const ContextMap &context) const;

QString filterString() const;
QString debugString() const;

private:
Filter(ExpressionPtr expression, Type returnType);

ExpressionPtr expression_;
Type returnType_;
};

} // namespace chatterino::filters
Loading