Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add column 'level' to display which field is core/extended #149

Merged
merged 7 commits into from
Oct 29, 2018
Merged

Add column 'level' to display which field is core/extended #149

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ee482f8
Improve readability in use-cases.py
Oct 24, 2018
cb1625a
Add column 'level' to display what's core/extended. Get rid of phase
Oct 25, 2018
6586583
Generated files based on previous change
Oct 25, 2018
2727317
Fix a few use case fields that should now be `keyword`
Oct 25, 2018
a75b41c
Generates markdown
Oct 26, 2018
e8ab9df
Add the missing column for examples in the Markdown tables
Oct 26, 2018
804928d
Move new 'level' column between description and type
Oct 26, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
376 changes: 188 additions & 188 deletions README.md
View file
Open in desktop

Large diffs are not rendered by default.

290 changes: 145 additions & 145 deletions schema.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,145 +1,145 @@
Field,Type,Phase,Example
@timestamp,date,0,2016-05-23T08:05:34.853Z
labels,object,0,"{'application': 'foo-bar', 'env': 'production'}"
message,text,0,Hello World
tags,keyword,0,"[""production"", ""env2""]"
agent.ephemeral_id,keyword,0,8a4f500f
agent.id,keyword,0,8a4f500d
agent.name,keyword,0,filebeat
agent.version,keyword,0,6.0.0-rc2
cloud.account.id,keyword,0,666777888999
cloud.availability_zone,keyword,0,us-east-1c
cloud.instance.id,keyword,0,i-1234567890abcdef0
cloud.instance.name,keyword,0,
cloud.machine.type,keyword,0,t2.medium
cloud.provider,keyword,0,ec2
cloud.region,keyword,0,us-east-1
container.id,keyword,0,
container.image.name,keyword,0,
container.image.tag,keyword,0,
container.labels,object,0,
container.name,keyword,0,
container.runtime,keyword,0,docker
destination.domain,keyword,0,
destination.hostname,keyword,0,
destination.ip,ip,0,
destination.mac,keyword,0,
destination.port,long,0,
destination.subdomain,keyword,0,
device.hostname,keyword,0,
device.ip,ip,0,
device.mac,keyword,0,
device.serial_number,keyword,0,
device.type,keyword,0,firewall
device.vendor,keyword,0,
device.version,keyword,0,
error.code,keyword,0,
error.id,keyword,0,
error.message,text,0,
event.action,keyword,0,reject
event.category,keyword,0,metrics
event.created,date,0,
event.dataset,keyword,0,stats
event.duration,long,0,
event.hash,keyword,0,123456789012345678901234567890ABCD
event.id,keyword,0,8a4f500d
event.module,keyword,0,mysql
event.original,keyword,0,Sep 19 08:26:10 host CEF:0|Security| threatmanager|1.0|100| worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2spt=1232
event.risk_score,float,0,
event.risk_score_norm,float,0,
event.severity,long,0,7
event.type,keyword,0,nginx-stats-metrics
event.version,keyword,0,0.1.0
file.ctime,date,0,
file.device,keyword,0,
file.extension,keyword,0,png
file.gid,keyword,0,
file.group,keyword,0,
file.inode,keyword,0,
file.mode,keyword,0,416
file.mtime,date,0,
file.owner,keyword,0,
file.path,keyword,0,
file.size,long,0,
file.target_path,keyword,0,
file.type,keyword,0,
file.uid,keyword,0,
geo.city_name,keyword,0,
geo.continent_name,keyword,0,
geo.country_iso_code,keyword,0,
geo.location,geo_point,0,
geo.region_name,keyword,0,
host.architecture,keyword,0,x86_64
host.hostname,keyword,0,
host.id,keyword,0,
host.ip,ip,0,
host.mac,keyword,0,
host.os.family,keyword,0,debian
host.os.name,keyword,0,Mac OS X
host.os.platform,keyword,0,darwin
host.os.version,keyword,0,10.12.6
host.type,keyword,0,
http.request.method,keyword,0,"GET, POST, PUT"
http.response.body,keyword,0,Hello world
http.response.status_code,long,0,404
http.version,keyword,0,1.1
log.level,keyword,0,ERR
log.original,keyword,0,Sep 19 08:26:10 localhost My log
network.direction,keyword,0,inbound
network.forwarded_ip,ip,0,192.1.1.2
network.inbound.bytes,long,0,184
network.inbound.packets,long,0,12
network.name,keyword,0,Guest Wifi
network.outbound.bytes,long,0,184
network.outbound.packets,long,0,12
network.protocol,keyword,0,http
network.total.bytes,long,0,368
network.total.packets,long,0,24
organization.id,keyword,0,
organization.name,keyword,0,
os.family,keyword,0,debian
os.kernel,keyword,0,4.4.0-112-generic
os.name,keyword,0,Mac OS X
os.platform,keyword,0,darwin
os.version,keyword,0,10.12.6-rc2
process.args,keyword,0,"['-l', 'user', '10.0.0.16']"
process.name,keyword,0,ssh
process.pid,long,0,
process.ppid,long,0,
process.title,keyword,0,
service.ephemeral_id,keyword,0,8a4f500f
service.id,keyword,0,d37e5ebfe0ae6c4972dbe9f0174a1637bb8247f6
service.name,keyword,0,elasticsearch
service.state,keyword,0,
service.type,keyword,0,
service.version,keyword,0,3.2.4
source.domain,keyword,0,
source.hostname,keyword,0,
source.ip,ip,0,
source.mac,keyword,0,
source.port,long,0,
source.subdomain,keyword,0,
url.fragment,keyword,0,
url.hostname,keyword,0,elastic.co
url.href,keyword,0,https://elastic.co:443/search?q=elasticsearch#top
url.password,keyword,0,
url.path,keyword,0,
url.port,integer,0,443
url.query,keyword,0,
url.scheme,keyword,0,https
url.username,keyword,0,
user.email,keyword,0,
user.hash,keyword,0,
user.id,keyword,0,
user.name,keyword,0,
user_agent.device,keyword,0,
user_agent.major,long,0,
user_agent.minor,long,0,
user_agent.name,keyword,0,Chrome
user_agent.original,keyword,0,
user_agent.os.major,long,0,
user_agent.os.minor,long,0,
user_agent.os.name,keyword,0,
user_agent.os.version,keyword,0,
user_agent.patch,keyword,0,
user_agent.version,keyword,0,
Field,Type,Level,Example
@timestamp,date,core,2016-05-23T08:05:34.853Z
labels,object,core,"{'application': 'foo-bar', 'env': 'production'}"
message,text,core,Hello World
tags,keyword,core,"[""production"", ""env2""]"
agent.ephemeral_id,keyword,extended,8a4f500f
agent.id,keyword,core,8a4f500d
agent.name,keyword,core,filebeat
agent.version,keyword,core,6.0.0-rc2
cloud.account.id,keyword,extended,666777888999
cloud.availability_zone,keyword,extended,us-east-1c
cloud.instance.id,keyword,extended,i-1234567890abcdef0
cloud.instance.name,keyword,extended,
cloud.machine.type,keyword,extended,t2.medium
cloud.provider,keyword,extended,ec2
cloud.region,keyword,extended,us-east-1
container.id,keyword,core,
container.image.name,keyword,extended,
container.image.tag,keyword,extended,
container.labels,object,extended,
container.name,keyword,extended,
container.runtime,keyword,extended,docker
destination.domain,keyword,core,
destination.hostname,keyword,core,
destination.ip,ip,core,
destination.mac,keyword,core,
destination.port,long,core,
destination.subdomain,keyword,core,
device.hostname,keyword,core,
device.ip,ip,core,
device.mac,keyword,core,
device.serial_number,keyword,extended,
device.type,keyword,core,firewall
device.vendor,keyword,core,
device.version,keyword,core,
error.code,keyword,core,
error.id,keyword,core,
error.message,text,core,
event.action,keyword,core,reject
event.category,keyword,core,metrics
event.created,date,core,
event.dataset,keyword,core,stats
event.duration,long,core,
event.hash,keyword,extended,123456789012345678901234567890ABCD
event.id,keyword,core,8a4f500d
event.module,keyword,core,mysql
event.original,keyword,core,Sep 19 08:26:10 host CEF:0|Security| threatmanager|1.0|100| worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2spt=1232
event.risk_score,float,core,
event.risk_score_norm,float,extended,
event.severity,long,core,7
event.type,keyword,core,nginx-stats-metrics
event.version,keyword,core,0.1.0
file.ctime,date,extended,
file.device,keyword,extended,
file.extension,keyword,extended,png
file.gid,keyword,extended,
file.group,keyword,extended,
file.inode,keyword,extended,
file.mode,keyword,extended,416
file.mtime,date,extended,
file.owner,keyword,extended,
file.path,keyword,extended,
file.size,long,extended,
file.target_path,keyword,extended,
file.type,keyword,extended,
file.uid,keyword,extended,
geo.city_name,keyword,core,
geo.continent_name,keyword,core,
geo.country_iso_code,keyword,core,
geo.location,geo_point,core,
geo.region_name,keyword,core,
host.architecture,keyword,core,x86_64
host.hostname,keyword,core,
host.id,keyword,core,
host.ip,ip,core,
host.mac,keyword,core,
host.os.family,keyword,extended,debian
host.os.name,keyword,extended,Mac OS X
host.os.platform,keyword,extended,darwin
host.os.version,keyword,extended,10.12.6
host.type,keyword,core,
http.request.method,keyword,extended,"GET, POST, PUT"
http.response.body,keyword,extended,Hello world
http.response.status_code,long,extended,404
http.version,keyword,extended,1.1
log.level,keyword,core,ERR
log.original,keyword,core,Sep 19 08:26:10 localhost My log
network.direction,keyword,core,inbound
network.forwarded_ip,ip,core,192.1.1.2
network.inbound.bytes,long,core,184
network.inbound.packets,long,core,12
network.name,keyword,extended,Guest Wifi
network.outbound.bytes,long,core,184
network.outbound.packets,long,core,12
network.protocol,keyword,core,http
network.total.bytes,long,core,368
network.total.packets,long,core,24
organization.id,keyword,extended,
organization.name,keyword,extended,
os.family,keyword,extended,debian
os.kernel,keyword,extended,4.4.0-112-generic
os.name,keyword,extended,Mac OS X
os.platform,keyword,extended,darwin
os.version,keyword,extended,10.12.6-rc2
process.args,keyword,extended,"['-l', 'user', '10.0.0.16']"
process.name,keyword,extended,ssh
process.pid,long,core,
process.ppid,long,extended,
process.title,keyword,extended,
service.ephemeral_id,keyword,extended,8a4f500f
service.id,keyword,core,d37e5ebfe0ae6c4972dbe9f0174a1637bb8247f6
service.name,keyword,core,elasticsearch
service.state,keyword,core,
service.type,keyword,core,
service.version,keyword,core,3.2.4
source.domain,keyword,core,
source.hostname,keyword,core,
source.ip,ip,core,
source.mac,keyword,core,
source.port,long,core,
source.subdomain,keyword,core,
url.fragment,keyword,extended,
url.hostname,keyword,extended,elastic.co
url.href,keyword,extended,https://elastic.co:443/search?q=elasticsearch#top
url.password,keyword,extended,
url.path,keyword,extended,
url.port,integer,extended,443
url.query,keyword,extended,
url.scheme,keyword,extended,https
url.username,keyword,extended,
user.email,keyword,extended,
user.hash,keyword,extended,
user.id,keyword,core,
user.name,keyword,core,
user_agent.device,keyword,extended,
user_agent.major,long,extended,
user_agent.minor,long,extended,
user_agent.name,keyword,extended,Chrome
user_agent.original,keyword,extended,
user_agent.os.major,long,extended,
user_agent.os.minor,long,extended,
user_agent.os.name,keyword,extended,
user_agent.os.version,keyword,extended,
user_agent.patch,keyword,extended,
user_agent.version,keyword,extended,
13 changes: 5 additions & 8 deletions scripts/helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ def clean_fields(fields, prefix, group):
if prefix != "":
field["name"] = prefix + "." + field["name"]

if 'phase' not in field.keys():
field["phase"] = 0
if 'level' not in field.keys():
field["level"] = '(use case)'

if 'group' not in field.keys():
# If no group set, set parent group
Expand All @@ -69,9 +69,6 @@ def clean_fields(fields, prefix, group):
# multi fields always have a prefix
f["name"] = field["name"] + "." + f["name"]

if 'phase' not in f.keys():
f["phase"] = 0

if 'group' not in f.keys():
# If no group set, set parent group
f["group"] = group
Expand Down Expand Up @@ -117,10 +114,10 @@ def get_markdown_row(field, link, multi_field):

# If link is true, it link to the anchor is provided. This is used for the use-cases
if link and ecs:
return '| [{}]({}#{}) | {} | {} | {} | {} |\n'.format(show_name, link, field["name"], description, field["type"], multi_field, example)
return '| [{}]({}#{}) | {} | {} | {} | {} |\n'.format(show_name, link, field["name"], field["level"], description, field["type"], multi_field, example)

# By default a anchor is attached to the name
return '| <a name="{}"></a>{} | {} | {} | {} | {} |\n'.format(field["name"], show_name, description, field["type"], multi_field, example)
return '| <a name="{}"></a>{} | {} | {} | {} | {} |\n'.format(field["name"], show_name, field["level"], description, field["type"], multi_field, example)


def get_schema():
Expand All @@ -137,7 +134,7 @@ def get_markdown_table(namespace, title_prefix="##", link=False):
# Replaces one newlines with two as otherwise double newlines do not show up in markdown
output += namespace["description"].replace("\n", "\n\n") + "\n"

titles = ["Field", "Description", "Type", "Multi Field", "Example"]
titles = ["Field", "Level", "Description", "Type", "Multi Field", "Example"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would put the level after the Type.

As we don't have multi fields anymore, we can remove the multi field part? Or should we keep it just in case. Not related to this PR but stumbled over it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree level at column 2 is not ideal.

However putting level after type puts it between type and multi-field, which are closely related. I'll try with level just before type instead, and let's see how this looks.


for title in titles:
output += "| {} ".format(title)
Expand Down
8 changes: 5 additions & 3 deletions scripts/schemas.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ def create_csv(fields, file):
if sys.version_info >= (3, 0):
open_mode = "w"

# Create markdown schema output file
# Output schema to csv
with open(file, open_mode) as csvfile:
schema_writer = csv.writer(csvfile,
delimiter=',',
quoting=csv.QUOTE_MINIMAL,
lineterminator='\n')
schema_writer.writerow(["Field", "Type", "Phase", "Example"])
schema_writer.writerow(["Field", "Type", "Level", "Example"])

for namespace in fields:
if len(namespace["fields"]) == 0:
Expand All @@ -31,7 +31,7 @@ def create_csv(fields, file):

# Print fields into a table
for field in namespaceFields:
schema_writer.writerow([field["name"], field["type"], field["phase"], field["example"]])
schema_writer.writerow([field["name"], field["type"], field["level"], field["example"]])


def create_markdown(fields, file):
Expand Down Expand Up @@ -95,12 +95,14 @@ def check_fields(fields):

check_fields(sortedNamespaces)

# Generates html for README
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It creates Markdown not HTML :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha true :-)

if args.stdout == "true":
groups = [1, 2, 3]
f_fields = filtered_fields(sortedNamespaces, groups)
# Print to stdout
print(create_markdown_string(f_fields))

# Generates schema.csv
else:
groups = [1, 2, 3]
f_fields = filtered_fields(sortedNamespaces, groups)
Expand Down
Loading