-
Notifications
You must be signed in to change notification settings - Fork 0
/
rss.xml
104 lines (95 loc) · 27.8 KB
/
rss.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Lia Nemeth]]></title><description><![CDATA[blog about software, music and games.]]></description><link>https://lianemeth.com</link><generator>GatsbyJS</generator><lastBuildDate>Mon, 13 Jun 2022 22:09:53 GMT</lastBuildDate><item><title><![CDATA[Live Improvisation on Twitch]]></title><description><![CDATA[I have left this blog abandoned for a while. I will try to populate more often with some small updates on what I’ve been up to. Lately I…]]></description><link>https://lianemeth.com/live-improvisation-on-twitch/</link><guid isPermaLink="false">https://lianemeth.com/live-improvisation-on-twitch/</guid><pubDate>Sun, 19 Jun 2022 00:00:00 GMT</pubDate><content:encoded><p>I have left this blog abandoned for a while. I will try to populate more often with some small updates on what I’ve been up to.</p>
<p>Lately I have been very dedicated to music on my free time, I play keyboards and used to be a professional musician prior to changing careers to Software Engineering.</p>
<p>I have been playing live improvisation on twitch on <a href="https://www.twitch.tv/frogsynth">https://www.twitch.tv/frogsynth</a> and it has been a great way to explore this side of me that had been less active in the last years. It is hard to define the music I make under a specific genre, but closest would be jazz fusion.</p>
<p>The format of live streaming is incredible for the tradition of fully improvised music. Improvisation has an ephemeral element that makes it special, the artwork created in an improvisation session is unique to the performance, and it is representative of that event, and of the connection between the artist and the audience. Live streaming make it possible to create a routine out of it, and create an audience looking forward to hear the new material being created and share that space with the performer. </p>
<p>Hope you give it a listen! I’m also working on an album that will soon be released!</p></content:encoded></item><item><title><![CDATA[High Signal Application Logging]]></title><description><![CDATA[Application logging is often put aside as lower priority. It is rare to see applications that treat logging as a first class element of…]]></description><link>https://lianemeth.com/high-signal-application-logging/</link><guid isPermaLink="false">https://lianemeth.com/high-signal-application-logging/</guid><pubDate>Thu, 24 Sep 2020 00:00:00 GMT</pubDate><content:encoded><p>Application logging is often put aside as lower priority. It is rare to see applications that treat logging as a first class element of their architecture, but logging done right can be a powerful tool on increasing the visibility of live environments. Good quality logs play a crucial role in creating alerts and visualizations. If your logs are informative and clear, it becomes easy to use them with tools for creating valuable dashboards. Tools such as <em>ELK</em>, <em>Splunk</em> or <em>Datadog</em> will easily allow you to do this, but this is a topic I can expand on in another blog post. Likewise, high signal logs can translate into high signal alerts that inform you about abnormal application behavior.</p>
<p>I consider logging to be essential for my workflow, but informative logs are also a key component on attack repudiation. I learned it from the threat modelling technique <a href="https://en.wikipedia.org/wiki/STRIDE_(security)">STRIDE</a> which is a famous mnemonic that lists six categories of threats. The <strong>R</strong> stands for repudiation, and it means the ability to assess that something has happened and what it can be done to fix it. For instance, a user can perform some sort of malicious or fraudulent activity. If this information is not logged somewhere, the application owner won’t be able to react, and might even entirely miss it.</p>
<h2>Formatting</h2>
<p>Logs should always use machine readable formats. This will allow the application to use a log management tool to its full potential. </p>
<p>Being able to aggregate logs, and search them by common keys will make it easier to scale application monitoring to multiple servers and services. </p>
<p>My favorite format for logging is <strong>JSON</strong>, while it is not the best for human readability, it’s supported in all log management services. </p>
<p>This example show how JSON logging works on Python:</p>
<h3>JSON logger</h3>
<div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">import</span> json
<span class="token keyword">import</span> logging
<span class="token keyword">import</span> time
<span class="token keyword">class</span> <span class="token class-name">LogEncoder</span><span class="token punctuation">(</span>json<span class="token punctuation">.</span>JSONEncoder<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">default</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> obj<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">if</span> <span class="token builtin">isinstance</span><span class="token punctuation">(</span>obj<span class="token punctuation">,</span> <span class="token builtin">set</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token builtin">tuple</span><span class="token punctuation">(</span>obj<span class="token punctuation">)</span>
<span class="token keyword">return</span> <span class="token builtin">super</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>default<span class="token punctuation">(</span>obj<span class="token punctuation">)</span>
<span class="token keyword">class</span> <span class="token class-name">JsonLogger</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> message<span class="token punctuation">,</span> <span class="token operator">**</span>kwargs<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>kwargs <span class="token operator">=</span> kwargs
<span class="token comment"># hard-coded fields can be configured here</span>
self<span class="token punctuation">.</span>kwargs<span class="token punctuation">[</span><span class="token string">'message'</span><span class="token punctuation">]</span> <span class="token operator">=</span> message
self<span class="token punctuation">.</span>kwargs<span class="token punctuation">[</span><span class="token string">'timestamp'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span>time<span class="token punctuation">.</span>time<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">__str__</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> LogEncoder<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>encode<span class="token punctuation">(</span>self<span class="token punctuation">.</span>kwargs<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">example</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
logging<span class="token punctuation">.</span>basicConfig<span class="token punctuation">(</span>level<span class="token operator">=</span>logging<span class="token punctuation">.</span>INFO<span class="token punctuation">,</span>
<span class="token builtin">format</span><span class="token operator">=</span><span class="token string">'%(message)s'</span><span class="token punctuation">)</span>
logging<span class="token punctuation">.</span>info<span class="token punctuation">(</span>JsonLogger<span class="token punctuation">(</span>
message<span class="token operator">=</span><span class="token string">'this has happened'</span><span class="token punctuation">,</span>
user_id<span class="token operator">=</span><span class="token number">12</span><span class="token punctuation">,</span>
order_number<span class="token operator">=</span><span class="token number">400</span><span class="token punctuation">,</span>
<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
example<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre></div>
<h3>Output</h3>
<div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json"><span class="token punctuation">{</span><span class="token property">"user_id"</span><span class="token operator">:</span> <span class="token number">12</span><span class="token punctuation">,</span> <span class="token property">"order_number"</span><span class="token operator">:</span> <span class="token number">400</span><span class="token punctuation">,</span> <span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"this has happened"</span><span class="token punctuation">,</span> <span class="token property">"timestamp"</span><span class="token operator">:</span> <span class="token number">1600709474</span><span class="token punctuation">}</span></code></pre></div>
<p>If you create logging policies where common fields are re-used accross services, you will be able to aggregate them through type and value. Based on the example above, application logs could look this:</p>
<div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json"><span class="token punctuation">{</span><span class="token property">"user_id"</span><span class="token operator">:</span> <span class="token number">12</span><span class="token punctuation">,</span> <span class="token property">"order_number"</span><span class="token operator">:</span> <span class="token number">400</span><span class="token punctuation">,</span> <span class="token property">"order_status"</span><span class="token operator">:</span> <span class="token string">"unpaid"</span><span class="token punctuation">,</span> <span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Order has been created"</span><span class="token punctuation">,</span> <span class="token property">"timestamp"</span><span class="token operator">:</span> <span class="token number">1600709474</span><span class="token punctuation">,</span> <span class="token property">"path"</span><span class="token operator">:</span> <span class="token string">"/create_order"</span><span class="token punctuation">}</span>
<span class="token punctuation">{</span><span class="token property">"user_id"</span><span class="token operator">:</span> <span class="token number">12</span><span class="token punctuation">,</span> <span class="token property">"order_number"</span><span class="token operator">:</span> <span class="token number">400</span><span class="token punctuation">,</span> <span class="token property">"order_status"</span><span class="token operator">:</span> <span class="token string">"paid"</span><span class="token punctuation">,</span> <span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Order has been succesfully paid"</span><span class="token punctuation">,</span> <span class="token property">"timestamp"</span><span class="token operator">:</span> <span class="token number">1600709480</span><span class="token punctuation">,</span> <span class="token property">"path"</span><span class="token operator">:</span> <span class="token string">"/complete_order"</span><span class="token punctuation">}</span>
<span class="token punctuation">{</span><span class="token property">"user_id"</span><span class="token operator">:</span> <span class="token number">12</span><span class="token punctuation">,</span> <span class="token property">"order_number"</span><span class="token operator">:</span> <span class="token number">401</span><span class="token punctuation">,</span> <span class="token property">"order_status"</span><span class="token operator">:</span> <span class="token string">"unpaid"</span><span class="token punctuation">,</span> <span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Order has been created"</span><span class="token punctuation">,</span> <span class="token property">"timestamp"</span><span class="token operator">:</span> <span class="token number">1600709481</span><span class="token punctuation">,</span> <span class="token property">"path"</span><span class="token operator">:</span> <span class="token string">"create_order"</span><span class="token punctuation">}</span>
<span class="token punctuation">{</span><span class="token property">"user_id"</span><span class="token operator">:</span> <span class="token number">12</span><span class="token punctuation">,</span> <span class="token property">"order_number"</span><span class="token operator">:</span> <span class="token number">401</span><span class="token punctuation">,</span> <span class="token property">"order_status"</span><span class="token operator">:</span> <span class="token string">"failed"</span><span class="token punctuation">,</span> <span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Card has been declined"</span><span class="token punctuation">,</span> <span class="token property">"timestamp"</span><span class="token operator">:</span> <span class="token number">1600709487</span><span class="token punctuation">,</span> <span class="token property">"path"</span><span class="token operator">:</span> <span class="token string">"complete_order"</span><span class="token punctuation">}</span></code></pre></div>
<p>Where we can distinguish a machine readable user history: <code class="language-text">User 12</code> has made two orders in a short timespan, <code class="language-text">order 400</code> has been completed but <code class="language-text">order 401</code> had failed because the card was declined.</p>
<p>On a log aggregation system, we can easily query this information. For instance, in <em>Datadog</em> the logs above could be queried such as
<code class="language-text">@order_status:failed AND @path:complete_order</code>
and that could be used to verify how often orders are having their payment declined.</p>
<p>The key information for the application should have its own fields, not just in the body of the message. Common fields should be agreed upon between all of the developers of the system, so they can be reused by multiple services or libraries. Documentation around these fields should be made available.</p>
<h2>Context</h2>
<p>High signal logging can be obtained if you strive for low volume of highly informative log messages.
Verbosity is not an issue. That being said, it shouldn’t be through the <strong>amount</strong> of messages that get logged, but on <strong>how much information</strong> each of these messages contains. Verbosity stops being an issue if it’s easy to select only the relevant information from the group of logs. Good formatting and proper use of log levelling helps on that. I will explain levelling on the next section.</p>
<p>Each message should include the maximum context possible, without including personal or sensitive information. Examples of relevant information can be ids of the database, status code, endpoints, HTTP methods, state, etc. Once context is added on their own fields, repeated patterns can be identified, and made into their own fields into the log format structure.</p>
<p><em>pro-tip</em>: have certain internal classes and objects have their own logger encoders that expose only the relevant information. For instance:</p>
<div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">User</span><span class="token punctuation">(</span>something<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">log_encode</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span>
<span class="token string">'user_id'</span><span class="token punctuation">:</span> self<span class="token punctuation">.</span><span class="token builtin">id</span><span class="token punctuation">,</span>
<span class="token string">'user_status'</span><span class="token punctuation">:</span> self<span class="token punctuation">.</span>status
<span class="token punctuation">}</span></code></pre></div>
<p>This shows a class where the objects can be passed to the logger, which then calls the appropriate method for that object. That would abstract away the responsibility of selecting fields to be logged to the logged objects, making it easy to make changes on their format and on keeping a common pattern accross all log data.</p>
<h2>Levels</h2>
<p>Knowing when to use the correct log levels on an application can make the difference between noisy logs that often get ignored vs. informative concise logs that can easily be leveraged to increase application observability. </p>
<h4>Debug</h4>
<p>Information that is useful for developers and testers, but would be too verbose in a production scenario. This can be more verbose than the usual application logging, but production environments should be configured to supress these.</p>
<h4>Info</h4>
<p>Information about succesfull operations that has happened in the system, should be very informative and full of contextual information. This should be shown in production environments, so it’s important to avoid excessive <code class="language-text">Info</code> level messages. As a rule of thumb, one atomic action should only result in one <code class="language-text">Info</code> level log.</p>
<h4>Warning</h4>
<p>Warning messages are unexpected behaviors that do not cause the application to crash or the flow of operations to be interrupted. It should be available in production environments and used with caution.</p>
<h4>Error</h4>
<p>An error that caused the operation to stop, but not crash. In a web environment, an action that causes a status code 500 usually should also emit an <code class="language-text">error</code> log.</p>
<h4>Fatal</h4>
<p>Actions that cause the application to entirely crash. These should be used only when very relevant. In a web application, these should immediately trigger alerts.</p>
<h2>Dont’s!</h2>
<p>It’s important to never log certain types of information. Logging of sensitive or personal identifying information (PII) can result in breaking compliance of certain regulations such as <strong>GDRP</strong>.
Some of the things that should never be logged are:</p>
<ul>
<li>Passport numbers</li>
<li>Social Security Numbers</li>
<li>Government issued IDs</li>
<li>Passwords</li>
<li>Driver Licenses</li>
<li>Credit Card Numbers</li>
<li>Personal addresses</li>
<li>Telephone numbers</li>
<li>Personal Addresses</li>
<li>Public user names</li>
</ul>
<p>Anything that can identify a user and their person or a corporation or expose their private information should not be exposed in a log.
Logs should be kept as anonymous as possible.</p>
<h2>Conclusion</h2>
<p>Treating logging as a first class citizen in your application can increase your ability to know what is exactly going on your environments, and to be able to automate alerting and create visualizations based on that.
Thinking on logging early on during development helps create best practices that can be matured with the system.</p>
<p>I hope this article helps on you treating logging as an essential part of your application and how to leverage that power. Thanks for reading!</p></content:encoded></item></channel></rss>