[Packetbeat] New SIP protocol #21221

marc-gr · 2020-09-22T14:02:33Z

This is a re-opening of #7181 to fix merge issues.

TODO:

Parse VIA
Parse Contact
Parse Auth
Parse SDP body

Thanks to @tj8000rpm for the original work.

From the original PR:

Hi all. I implemented a new protocol SIP to packetbeat. #152
The SIP(Session Initiation Protocol) is a communications protocol for signaling and controlling multimedia communication sessions. SIP is used many VoIP applications at not only enterprise uses but also telecom careers.

SIP is text-base protocol like HTTP. But SIP has various unique features like :

SIP is server-client model, but it roles may changes call by call.
SIP is request-response model, but server may (usualy) reply many responses for one request.
There many requests and responses in one call.
It is not know when the call will end.

Therefore, I implemented with following plans.

Published for each SIP message(request or response)
Additional timestamp to make it possible to keep message order
Parse and store each header field and SDP body(option)
Received raw message is stored as text value(option)

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #21221 updated]
Start Time: 2020-10-06T09:39:55.819+0000
Duration: 44 min 20 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1079
Skipped	10
Total	1089

tj8000rpm · 2020-09-27T07:13:35Z

@marc-gr
I apologize my late reply.🙇
And Thank you for take over my Pull Request.I'll leave it to you. ❤️

dainperkins · 2020-09-28T14:34:23Z

@marc-gr - is the json in the readme the planned final output? (just wondering if ECS normalization is handled later, or if it should be worked in at this level?)

marc-gr · 2020-09-28T14:47:49Z

@marc-gr - is the json in the readme the planned final output? (just wondering if ECS normalization is handled later, or if it should be worked in at this level?)

This is the documentation coming from the original contribution, I will update all of it once we decide on mappings at elastic/ecs#420 and changes are done. Also will add some test pcaps to have some examples.

andrewkroh

Thanks for helping more the SIP PR along.

There's a PCAP file in the PR. Is there a test that uses it? If not, can we add a "system test" that uses the file.
This will need an asciidoc file. And SIP will need added in a few places.
- https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-overview.html
- https://www.elastic.co/guide/en/beats/packetbeat/current/configuration-protocols.html
SIP should be added to packetbeat config files.

packetbeat/protos/sip/sip_message.go

andrewkroh · 2020-09-29T11:27:51Z

packetbeat/protos/sip/sip_message.go

+	return headers, firstLines
+}
+
+// TODO:The procedure with Content-Encoding(RFC3261).


After merging this implementation let's create issues for the remaining TODOs and possibly document any limitations if they are likely to be encountered.

packetbeat/protos/sip/sip_message.go

packetbeat/protos/sip/sip_plugin.go

packetbeat/protos/sip/_meta/fields.yml

packetbeat/_meta/sample_outputs/sip.json

andrewkroh · 2020-10-02T13:51:04Z

For releasing this as beta, the code itself should log a notice about the use of beta software. We have the cfgwarn package that we normally use for this.

packetbeat/_meta/sample_outputs/sip.json

dainperkins · 2020-10-02T13:59:08Z

packetbeat/_meta/sample_outputs/sip.json

+    "sip.version": "2.0",
+    "source.ip": "192.168.1.2",
+    "source.port": 5060,
+    "status": "OK",


status should only be for sip.type "response", and I think should be sip.status?

Those are used by packetbeat and checked by system tests, we also have the sip specific ones.

dainperkins · 2020-10-02T14:02:21Z

packetbeat/_meta/sample_outputs/sip.json

+    "source.ip": "192.168.1.2",
+    "source.port": 5060,
+    "status": "OK",
+    "type": "sip"


does this need to be renamed or removed?

Same as previous one

marc-gr · 2020-10-05T15:54:40Z

After talking with @jamiehynds and since it is not very clear yet how we will want to deal with the multiple Via headers and the multiple SDP media types and attributes, for now I just added them as via.original and sdp.body.original to make them queryable, and to have some time to think about how we plan to deal with them once people starts using the proto more.

webmat · 2020-10-05T16:00:06Z

@marc-gr I haven't been able to follow all of @dainperkins' work on the SIP proposal for fields. I just skimmed it now.

The one thing that jumped out to me, that I'd like addressed in this PR is that all version fields should be keyword, not long/integer. I think even the event samples you have wouldn't work, since they have 2.0 in there. Also, we will soon transition all version fields to version (in the keyword family) which will enable other cool stuff.

marc-gr · 2020-10-05T16:01:35Z

Yeah I noticed that in the doc, but could not fix it because lack of permissions, in my code they are keywords already 👍

andrewstucki

Have some general questions/comments, don't know enough about the protocol itself to opine too much there, but overall looks fairly solid to me.

packetbeat/tests/system/golden/sip-expected.json

packetbeat/protos/sip/parser.go

new protocol:sip:update include list new protocol:sip:initial blank file DNSをベースとしてまずはDNSでやっている内容を解析開始・・・コメント付け中。読み進めた分更新パーサの実装を開始パーサをのデコーダをsipPlugin内に移設もろもろリクエスト、レスポンス判定を追加 SIPのパース, SDPのパース追加、パーサをいろいろばらばらに。醜い・・・ヘッダパース系メソッドをsipMessageのメンバに変更バッファリングの仕組みのプロトタイプ作成ファイル分割、オブジェクト毎にファイルを分割。その他実装を進めているところ。とちゅう publish methodの実装一部エラーハンドリングを追加 TODO更新とデータ構造を追加ヘッダ処理諸々追加,途中 README TODO更新 ' 実働確認用に追加フィールド名にsip.を付与、unixtimenanoをフィールドに追加(デフォルトのtimestampだとSIP信号を並び替えるのに精度不足なため。) フィールド命名規則を更新 Added english description change field name align the indents fields.yml update about timestamp テストケース追加 added testcase add testcases add testcases and fixed some bug cases add test cases add testcase parseSIPHeaders fixed bug cases. comment and refactoring update testcases add monitoring element named 'sip.message_ignored' move publish function call from expireBuffer to callback function when buffer expired. add testcase, bufferExpire remove unnecessary pkg add testcase at publish method add, edit and migrate test cases modify time duration change timer code remove fragmneted process translate comments add linux amd64 binary Comments translated update informations add windows bin update TODO list add no mandantory header parse check Add compact-form test case Add compact-form test case Add compact-form test case Add compact-form test case support compact form TODO list update add sip uri parser add detail mode add binary remove unnecessary file bug fix:broken when response parse in detail mode bug fix:detail mode modify detail mode modify detail mode Update Readme about compact form and parse detail sip header and request-uri Update Readme about compact form and parse detail sip header and request-uri Update Readme about compact form and parse detail sip header and request-uri Update Readme about compact form and parse detail sip header and request-uri expand config parsing options edit variable names arrangement and add text README file refine Readme message update Readme text update Readme on Configuration Go coding style was checked with golint Erase duplicate field in field.yml, move src and dst fields into sip filed Update docs, fields.asciidoc Update docs, fields.asciidoc

CHANGELOG.next.asciidoc

* new protocol:sip:Make a directory and Readme file. new protocol:sip:update include list new protocol:sip:initial blank file DNSをベースとしてまずはDNSでやっている内容を解析開始・・・コメント付け中。読み進めた分更新パーサの実装を開始パーサをのデコーダをsipPlugin内に移設もろもろリクエスト、レスポンス判定を追加 SIPのパース, SDPのパース追加、パーサをいろいろばらばらに。醜い・・・ヘッダパース系メソッドをsipMessageのメンバに変更バッファリングの仕組みのプロトタイプ作成ファイル分割、オブジェクト毎にファイルを分割。その他実装を進めているところ。とちゅう publish methodの実装一部エラーハンドリングを追加 TODO更新とデータ構造を追加ヘッダ処理諸々追加,途中 README TODO更新 ' 実働確認用に追加フィールド名にsip.を付与、unixtimenanoをフィールドに追加(デフォルトのtimestampだとSIP信号を並び替えるのに精度不足なため。) フィールド命名規則を更新 Added english description change field name align the indents fields.yml update about timestamp テストケース追加 added testcase add testcases add testcases and fixed some bug cases add test cases add testcase parseSIPHeaders fixed bug cases. comment and refactoring update testcases add monitoring element named 'sip.message_ignored' move publish function call from expireBuffer to callback function when buffer expired. add testcase, bufferExpire remove unnecessary pkg add testcase at publish method add, edit and migrate test cases modify time duration change timer code remove fragmneted process translate comments add linux amd64 binary Comments translated update informations add windows bin update TODO list add no mandantory header parse check Add compact-form test case Add compact-form test case Add compact-form test case Add compact-form test case support compact form TODO list update add sip uri parser add detail mode add binary remove unnecessary file bug fix:broken when response parse in detail mode bug fix:detail mode modify detail mode modify detail mode Update Readme about compact form and parse detail sip header and request-uri Update Readme about compact form and parse detail sip header and request-uri Update Readme about compact form and parse detail sip header and request-uri Update Readme about compact form and parse detail sip header and request-uri expand config parsing options edit variable names arrangement and add text README file refine Readme message update Readme text update Readme on Configuration Go coding style was checked with golint Erase duplicate field in field.yml, move src and dst fields into sip filed Update docs, fields.asciidoc Update docs, fields.asciidoc * Fixes and style changes * Refactor to be more similar to http parser and add system tests * Add event action * Add related fields * Update fields and docs * Add sip to docs * Add beta warning * Parse SDP, Contact, Via and auth * Add suggestions Co-authored-by: tj8000rpm <[email protected]> (cherry picked from commit 0dd2428)

* upstream/master: [CI] Setup git config globally (elastic#21562) docs: update generate_fields_docs.py (elastic#21359) Add support for additional fields from V2 ALB logs (elastic#21540) Move Prometheus query & remote_write to GA (elastic#21507) feat: add a new step to run the e2e tests for certain parts of Beats (elastic#21100) [Elastic Agent] Add elastic agent ID and version to events from filebeat and metricbeat. (elastic#21543) Release cloudfoundry input and processor as GA (elastic#21525) [Packetbeat] New SIP protocol (elastic#21221) [Filebeat][New Module] Add support for Microsoft MTP / 365 Defender (elastic#21446) [Beats][pytest] Asserting if filebeat logs include errors (elastic#20999) junipersrx-module initial release (elastic#20017) Add a persistent cache for cloudfoundry metadata based on badger (elastic#20775) Add missing changelog entry for cisco umbrella (elastic#21550) [Elastic Agent] Add upgrade CLI to initiate upgrade of Agent locally (elastic#21425) Enable filestream input (elastic#21533) Add filestream input reader (elastic#21481) [CI] fix 'no matches found within 10000' (elastic#21466) Fix billing.go aws.GetStartTimeEndTime (elastic#21531)

marc-gr added enhancement in progress Pull request is currently in progress. Packetbeat Team:SIEM v7.10.0 labels Sep 22, 2020

marc-gr requested a review from a team as a code owner September 22, 2020 14:02

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 22, 2020

marc-gr mentioned this pull request Sep 22, 2020

New protocol support: SIP #7181

Closed

marc-gr force-pushed the sip_protocol branch from 9489092 to eb64515 Compare September 22, 2020 14:23

ebeahan mentioned this pull request Sep 22, 2020

SIP package elastic/ecs#420

Closed

andrewkroh reviewed Sep 29, 2020

View reviewed changes

marc-gr force-pushed the sip_protocol branch from eb64515 to b38391e Compare October 2, 2020 11:48

dainperkins reviewed Oct 2, 2020

View reviewed changes

packetbeat/_meta/sample_outputs/sip.json Outdated Show resolved Hide resolved

dainperkins reviewed Oct 2, 2020

View reviewed changes

marc-gr force-pushed the sip_protocol branch from b6e1baa to 60a272e Compare October 5, 2020 15:38

andrewstucki reviewed Oct 5, 2020

View reviewed changes

tj8000rpm and others added 3 commits October 5, 2020 18:27

Fixes and style changes

97b7682

Refactor to be more similar to http parser and add system tests

36762da

marc-gr added 6 commits October 5, 2020 18:27

Add event action

dad6f7d

Add related fields

8c33879

Update fields and docs

845d5a6

Add sip to docs

0392cdd

Add beta warning

4c1704d

Parse SDP, Contact, Via and auth

62b1360

marc-gr force-pushed the sip_protocol branch 2 times, most recently from ccb0029 to b9e4bb7 Compare October 5, 2020 16:33

marc-gr requested a review from andrewstucki October 6, 2020 09:35

adriansr approved these changes Oct 6, 2020

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

Add suggestions

eeccf0e

marc-gr force-pushed the sip_protocol branch from b9e4bb7 to eeccf0e Compare October 6, 2020 09:39

marc-gr merged commit 0dd2428 into elastic:master Oct 6, 2020

marc-gr deleted the sip_protocol branch October 6, 2020 10:24

marc-gr mentioned this pull request Oct 6, 2020

Cherry-pick #21221 to 7.x: [Packetbeat] New SIP protocol #21557

Merged

4 tasks

jamiehynds mentioned this pull request Apr 19, 2021

New protocol support: SIP #152

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Packetbeat] New SIP protocol #21221

[Packetbeat] New SIP protocol #21221

marc-gr commented Sep 22, 2020 •

edited

Loading

elasticmachine commented Sep 22, 2020

elasticmachine commented Sep 22, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

tj8000rpm commented Sep 27, 2020

dainperkins commented Sep 28, 2020

marc-gr commented Sep 28, 2020

andrewkroh left a comment

andrewkroh Sep 29, 2020

andrewkroh commented Oct 2, 2020

dainperkins Oct 2, 2020 •

edited

Loading

marc-gr Oct 5, 2020

dainperkins Oct 2, 2020

marc-gr Oct 5, 2020

marc-gr commented Oct 5, 2020

webmat commented Oct 5, 2020

marc-gr commented Oct 5, 2020 •

edited

Loading

andrewstucki left a comment

[Packetbeat] New SIP protocol #21221

[Packetbeat] New SIP protocol #21221

Conversation

marc-gr commented Sep 22, 2020 • edited Loading

elasticmachine commented Sep 22, 2020

elasticmachine commented Sep 22, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

tj8000rpm commented Sep 27, 2020

dainperkins commented Sep 28, 2020

marc-gr commented Sep 28, 2020

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh Sep 29, 2020

Choose a reason for hiding this comment

andrewkroh commented Oct 2, 2020

dainperkins Oct 2, 2020 • edited Loading

Choose a reason for hiding this comment

marc-gr Oct 5, 2020

Choose a reason for hiding this comment

dainperkins Oct 2, 2020

Choose a reason for hiding this comment

marc-gr Oct 5, 2020

Choose a reason for hiding this comment

marc-gr commented Oct 5, 2020

webmat commented Oct 5, 2020

marc-gr commented Oct 5, 2020 • edited Loading

andrewstucki left a comment

Choose a reason for hiding this comment

marc-gr commented Sep 22, 2020 •

edited

Loading

elasticmachine commented Sep 22, 2020 •

edited by jenkins-beats-ci bot

Loading

dainperkins Oct 2, 2020 •

edited

Loading

marc-gr commented Oct 5, 2020 •

edited

Loading