Serious Security Vulnerabilities / Project Abandoned? #104
Comments
|
Any other CVE's to be aware of? Most of the package's / pip's needed for trunk-player can be safely upgraded on the minor version scale without having to fight about with code changes. Things like django core can only be upgraded to version 1.11.29 - granted 1.x series was EOL'd many years ago, with security / lts ending a year and a half ago. |
|
@bctrainers Im sure there is a handful on the packages side, if I have time I'll go through and check https://snyk.io/. I was thinking about forking this to a docker centric version and adding federation across systems |
|
@bctrainers These are the Vulnerable packages as of 9/11/2021 EDIT: THIS DOESNT INCLUDE SUB-DEPENDENCIES Vulnerable packages:
|
|
Yea sadly it's pretty abandoned. As my area went 100% encrypted a few years back and I have no used it in that long, it's been hard to dedicate time on something you just don't use. Always happy to see the few pull requests when they come. I guess enough, it's clear i'm the only one and as i'm not actively doing any development or upgrades I should just bite the bullet and call an official end to this. ;( |
|
Well if I have the time Im going to try to fork it and overhaul it a bit, Any thoughts on implmenting site federation from multiple systems? |
|
@dreinhold would you be willing to provide a breakdown of how this app works so I can rip and replace old libs with the newer Libs I am familiar w/ Django and celery but not Daphne and asgi/Websockets; However Im sure I can figure out whats happening w/ enough time First issue w/ Django 3 is Channels 2 depreciated Group() |
|
Yeah I think channels is the only sticking point, i think the rest is going to be minor changes. |
|
Ok, So from what I've gathered so far on From my understanding changing it to (see below*) should be in spec with channels 2 Then as for asgi.py it needs to be closer to (see below**); where Im getting a little lost is the URL's, does it currently send WS over specific URLs and if so where are those defined? Following that, the consumers need to be converted to a class. closer to (see below ***). Are you able to ELI5 what these functions (ws_connect, ws_receive, ws_disconnect ) are doing in radio.consumers.py Also by chance are you able to provide any guidance on converting these consumers per https://channels.readthedocs.io/en/latest/topics/consumers.html#basic-layout consumers.py ***class RadioConsumer(WebsocketConsumer):
def connect(self):
self.
self.room_name = self.scope['url_route']['kwargs']['room_name']
self.room_group_name = 'chat_%s' % self.room_name
# Join room group
async_to_sync(self.channel_layer.group_add)(
self.room_group_name,
self.channel_name
)
self.accept()
def disconnect(self, close_code):
# Leave room group
async_to_sync(self.channel_layer.group_discard)(
self.room_group_name,
self.channel_name
)
# Receive message from WebSocket
def receive(self, text_data):
text_data_json = json.loads(text_data)
message = text_data_json['message']
# Send message to room group
async_to_sync(self.channel_layer.group_send)(
self.room_group_name,
{
'type': 'chat_message',
'message': message
}
)ASGI.py **import os
from django.conf.urls import url
from django.core.asgi import get_asgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "trunk_player.settings")
django_asgi_app = get_asgi_application()
from channels.auth import AuthMiddlewareStack
from channels.routing import ProtocolTypeRouter, URLRouter
from radio.consumers import RadioConsumer
application = ProtocolTypeRouter({
"http": django_asgi_app,
# WebSocket chat handler
"websocket": AuthMiddlewareStack(
URLRouter([
url(r"^chat/admin/$", AdminChatConsumer.as_asgi()),
url(r"^chat/$", PublicChatConsumer.as_asgi()),
])
),
})send_mesg *@receiver(post_save, sender=Transmission, dispatch_uid="send_mesg")
def send_mesg(sender, instance, **kwargs):
from channels.layers import get_channel_layer
channel_layer = get_channel_layer()
#log.debug('Hit post save()')
#log.debug('DATA %s', json.dumps(instance.as_dict()))
#log.error('DATA %s', json.dumps(instance.as_dict()))
tg = TalkGroup.objects.get(pk=instance.talkgroup_info.pk)
tg.last_transmission = timezone.now()
tg.save()
groups = tg.scanlist_set.all()
for g in groups:
async_to_sync(channel_layer.group_send)('livecall-scan-'+g.slug, {'text': json.dumps(instance.as_dict())})
#Group('livecall-scan-'+g.slug, ).send({'text': json.dumps(instance.as_dict())})
async_to_sync(channel_layer.group_send)('livecall-tg-' + tg.slug, {'text': json.dumps(instance.as_dict())})
#Group('livecall-tg-' + tg.slug, ).send({'text': json.dumps(instance.as_dict())})
# Send notification to default group all the time
async_to_sync(channel_layer.group_send)('livecall-scan-default', {'text': json.dumps(instance.as_dict())})
#Group('livecall-scan-default').send({'text': json.dumps(instance.as_dict())}) |
|
The WS connections are first passed to the routing.py where if it is a connect it is passed to consumers.py to the ws_connect() method. In that method it gets the full url path to get the tg/scan group the user is on to register the connection to that group in channels. The when a new transmission is added it looks up all the scan groups that tg is in and fires the messaged into all those groups to let the browser js know there is an updated transmission for a tg they are watching. |
|
@dreinhold Thank you! I think im starting to get a hold on whats going on, If you have time can you please review commits below, If im not mistaken should implement channels v2 Edit: btw https://github.com/MaxwellDPS/trunk-player/tree/74ef5acc27a3d28ae3549655552c1a04954a2a00 |
|
Likely has some bugs still but Channel2 is implemented on https://github.com/MaxwellDPS/trunk-player/tree/beta Edit: Scan groups connect |
|
So after I do some transmission testing, I'll submit a pull with the upgrade to Channels v2 and that make it so Django and all other deps are more up-to date. This should resolve all of the above vulns above ^. Then I think it should be easier to stay up on this project. and hopefully save Trunk-Player from the grave I do plan to fork this with an emphasis on a new UI, Multi-TrunkPlayer Federation, removal of Payment features, I would love to get some advise on all of the above! |
|
You are the man, I was sad to say it had died and you just went a brought it back to life. Maybe start a new issue for each of the ideas. I'm not sure what the federation you are looking for, like multiple trunk player sites working together, or multiple recording sites feeding trunk player? |
|
You're the man, you did the heavy lifting on this one lol. Ill try to get that PR out Tomorrow morning! And as for Federation I have a mix of use cases for both multiple trunk player sites working together, and multiple recording sites feeding trunk player |
|
@dreinhold Shoot, Just realized that the Django Jump from 2-3 Removes legacy python2 support which will break Pinax Stripe, Do you want me to try and fix that or PR w/o Plans and Pinax |
|
It looks like Pinax Stripe is also a project on deaths door, so no need to bring it with us. I am good dropping support for it (anyone actively using it we will need to find an alternate) |
|
Yeah, this is gonna have some breaking changes to the DB... Hmmm, may need to figure out a migration script for legacy users |
|
I tagged the current as "django-1.x" so once we pull in your changes if someone is using something that breaks, makes it easier to for them to get to the old. |
|
what are you seeing break in the DB. |
|
With the removal of Pinax stripe and plans it throws |
|
Eww, yeah. Thinking about this. |
|
Went ahead and synced my fork of TrunkPlayer, it would appear there are many more security vulnerabilities than I thought. https://github.com/bctrainers/trunk-player/security/dependabot Keep in mind my fork does have numerous changes that probably break stock-TrunkPlayer that @dreinhold has made. |
|
@dreinhold editing the 0038 migration fixed the migrations |
|
@dreinhold Just ssubmitted PR #105 likely want to test it a bit but should be GTG. You may want to check the references to Pinax stripe |
Upon my initial review, things look good - but have not tested it on a machine yet. When I get a moment this afternoon/evening, I will test it on a machine to see if anything explodes ( figuratively speaking :^) ). Are there any key points to note, or upgrade process notes from the current git version and your pending pull request? |
|
So @bctrainers I need some feedback on the migrations on a legacy system. Just spun up an new one and it works flawlessly with a clean DB. As for notes:
|
|
@bctrainers another quirk you may have to nuke you venv and recreate it |
|
@dreinhold are the workers being used, because after the update you have to pass the channels that it will work... so a few problems there lol, but it seems the websockets are connecting w/o them
|
|
The runworkers are getting usage on @dreinhold's current TrunkPlayer code via supervisor. I've 20 running on an install currently. They're handling end-client connections (WSS and HTTPS) from daphne via the Nginx stuff. |
Yes the runworkers changed from 1 to 2 (I don't fully understand what that means for us) |
Should this work? # -*- coding: utf-8 -*-
# Generated by Django 1.11.17 on 2019-01-04 22:55
from __future__ import unicode_literals
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('radio', '0062_siteoption_copyright_notice'),
]
operations = [
migrations.RemoveField(model_name='StripePlanMatrix', name="stripe_plan")
] |
Yeah per this
I think this means we are out of luck on the workers aside from maybe |
I would love to feed multiple recording sites into one trunk-player site :) This would be super useful for my site.. Thanks for bringing this project back to life @MaxwellDPS I personally would love to see this project kept alive and willing to help where I can.. Thanks Sonic |
|
Howdy everyone, Out of curiosity, looks like the webUI is JS based... Any react devs out there 😉👀 |
|
@dreinhold so I have been running the new fork in docker for a few days w/o runworker and so far I have not noticed any issues. I'd image a site with heavy load may have some delay... Not sure how to handle yet aside from rewriting for a unicast ish channel and js logic. May need to completely rework how the WS and JS interact, for the workers |
+1 on the idea of using react as the frontend. |
@dreinhold I have found away to accomplish this via a single uni-com WS channel, instead of diffrent ones we send all new messages out to that channel, then we can use JS for some magic. Then workers are useable too @bctrainers one problem is this is a security issue for TG restriction and should be resolved but not sure how yet... Expect a PR by EOD |
Edit: Slightly mitigated the security issue by limiting the WS info pushed to the bare minim. so in theory an and un authenticated / unauthorized user could see the TG name, ScanList Names, I can fix the unauthed user with template tags... Cant think of a great way we can obscure scan list names and TG names w/o using dynamic channel names... aside from asymmetric crypto and client IDs, but that seems like overkill |
|
Right now the channel is just a trigger for the page to re pull the api, so you really don't need to pass/leak any data. |
|
As you saw i created a django3 branch for all this until we can show both a new install, and an upgrade will work. Then we can make it master. We should start new issues at this point for new items/ideas, there is a Discussions thing in github now im not sure how it work but ill enable it, might be a good area where new ideas can be hashed out. Another big part to this is updating the documentation, we have the markup stuff in docs that gets built into the readthedocs site and the wiki. Need to look those over and edit as needed, for those who don't know django but have used TP would be a big help in updating the documents. |
|
A big thanks to @MaxwellDPS for just starting the conversation and for the updated code. I will be around, and still plan on reviewing the changes I can and jumping in where I can. Dylan |
|
While we're on the basis of a new version of TrunkPlayer, what are our current thoughts with supporting MySQL/MariaDB in addition to PostgreSQL? FWIW, pg13 works without much issues on TrunkPlayers current code. So that's always good. My apologies for not replying sooner on testing, real life has been getting in the way. :) |
|
Just wanted to say this is so great to see. I hope to also get some time to play!!!! |
@bctrainers @dreinhold correct me if Im wrong, but shouldn't setting the Django backend to mySQL work out of the box... may also need to install the pip package DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'trunk_player', # Database Name
'USER': 'trunk_player_user', # Database User Name
'PASSWORD': 'fake_password', # Database User Password
'HOST': 'localhost',
'PORT': '3306',
}
} |
|
Hey all, Just tested an upgrade of a clean install and @dreinhold the 0063_stablize.py migration crashes with a key error... I think we are ok to orphan Upgrade Steps:
|
|
FYI -- I run about 220 Recorders into one Trunk-Player instance. Scaling
is difficult, I had to separate out my database platform (postgres) onto
separate bare metal, and still run that hardware at about 75% CPU all of
the time with 15-30 active database sessions going at once.
The database is certainly the slowest part of the interface as you scale
up. Especially when you go into admin and try to create/add the scan lists
(something to do with the way it loads all of the talkgroups into the
selection dialog) will timeout *every time* for us (about 80K talkgroups)
We actually had to build a small instance and remove all the timeouts to be
able to do those kind of data operations (create/edit new scanlists); I'm
not sure how the query is assembled to build that table, but I'm assuming
there has to be some kind of a loop, with a single query that is executing
for every talkgroup. I think that query could be improved, if that's even
possible with Django.
I'm one of the people who attended Dylan's training sessions. I was working
on developing ideas and trying to learn how django works, but the hill was
too steep for me. I was working with Rachel O. Who suddenly and
unexpectedly passed away, and haven't had anyone else to work with since.
I don't really want the project to die, but as others have said there
hasn't been any development in a long time, and I don't currently have
enough knowledge to make the wide scale changes that I feel need to be made
to enhance the operation and bring new features.
I'm happy to participate in helping others.
…On Wed, Sep 15, 2021 at 9:03 PM Max Watermolen ***@***.***> wrote:
Hey all,
Just tested an upgrade of a clean install and @dreinhold
<https://github.com/dreinhold> the 0063_stablize.py migration crashes
with a key error... I think we are ok to orphan stripe_plan, we can
always write a postgresql command to clip it. If we remove that migration
the upgrade is smooth, but need some brave volunteers with backups ;). TBH
I broke my old site (Older Revision) but just connected the DB to the new
code and its been running smooth
Upgrade Steps:
1. Stop trunk player
2. cd trunk-player
3. git checkout django3
4. pip install -r requirements.txt
5. ./mange.py migrate
6. Restart Trunk player
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#104 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACSNSAS5J3GRSLBSSX2ML3LUCFF53ANCNFSM5D3DIIAA>
.
|
|
@kcwebby Wow, That sounds like a hell of system! 😄 So I can change the talk group fields to raw ID field, which means you'd need to know its ID in the DB but it will load a hell of a lot better, I think django has a UI even with that to help pick and loads in a separate window when needed. I can implement that as a setting in settings.py or via the env... TBH I want to try to make this as Mirco-service/cloud native as possible, so I will be putting a lot of focus towards docker deployments. I would be more than happy to help with getting docker instances stood up too as it seems to be way easier to scale and very reliable. Also Its a hell of a learning curve but with that big of a system @kcwebby Kubernetes could really make your life easier in the long term. |
|
@kcwebby may be a bit big but it should be snappy as heck... shouldnt break anything, you can test by and follow the instructions below, If it works well, I will merge it in. If not LMK and we can try the raw ID EDIT trunk-player/radio/admin.py class ScanListAdmin(admin.ModelAdmin):
form = ScanListAdminForm
save_as = True
save_on_top = TrueWITH THIS class ScanListAdmin(admin.ModelAdmin):
autocomplete_fields= ('talkgroups',)
|
Hey @dreinhold, not to call you out but, is there any active dev still happening on this project?
There are some serious concerns I have with this app, especially if its gonna be WAN accessible.
In the meantime Id seriously recommend no-one put this app on the internet
@dreinhold no shame if you cant maintain it, You created a great project and I love it! However Id seriously recommend you archive and label it with these warnings
Thanks
Max :)
The text was updated successfully, but these errors were encountered: