No recordings displayed on recordings screen #219
Comments
ghost
assigned 
|
Wow, great data gathering! And yes, the exception looks related, especially the line: Would you try this please (if you already haven't:) wget -d \
--header='Accept: application/json' \
--header='Accept-Encoding: gzip' \
--output-document=/tmp/JsonData.gz \
<yourBackendHostname>:6544/Dvr/GetRecordedList?Descending=trueYou should then be able to gunzip /tmp/JsonData.gz. The above will be very long single line that starts similar to this: {"ProgramList": {"StartIndex": "0", "Count": "246", "TotalAvailable": "246", "AsOf":...Just to save time, please mention if you build MythTV from source. Let's start with that and see what happens, thanks. |
|
@Speckto, note that 1.20.0 was released for beta testers today, 10/10/2013. |
|
Hi, Thanks for the quick response. I logged in using the link from the beta testers site and it redirected me to https://play.google.com/apps/testing/org.mythtv which was an error page stating the requested URL was not found. Potentially this is because I am not a member of google+ - I'm not really sure why that doesn't work. I'll fiddle with it later and see if I can get it working. I am running Gentoo so yes I have compiled from source via emerge. My mythtv details are: I ran the wget command - I had to remove the -d as wget was not compiled with debug support but the command worked anyway and I ended up with a 45K gzip. I uncompressed it without issue and ran iconv to validate it (I'm unsure how useful this is): Opening that in Gvim it gives a description for an episode in the series The 'e' on fiance in the above is actually accented (lost in my copy-paste). Gvim's goto takes me to the letter 'c' in fiance. Console vim won't display the accented e (or any other accented characters in other recordings) but gvim does. I checked the episode using mythfrontend and it displays the correctly accented character as well. I've uploaded the gzip here - not sure if you needed it as you didn't say: Now I can probably just delete the episode to fix this (assuming that is the only error which I somehow doubt; either way I'd prefer not to), but I'm guessing you might want some more diagnostics before I do anything like that given this displays in the front-end ok? Cheers. |
|
@Speckto, it would be great if you kept the episode(s) causing the I'll take a look at the .gz you posted. |
|
@Speckto, OK, here's a hex dump of the 1st instance: And, from Wikipedia, 0xe9 would fit: Which says that e9 should have three bytes where yours has only one So, where's it coming from? [answer, I don't know] Can you Here's an hd from my program table where the word que[space] @dmfrey, I found a comment while googling, could MAF be modified to |
|
The output of '\s' is: According to phpmyadmin, all my mythconverg tables are in "utf8_general_ci". I'm not sure if any of the above proves it or not (I've barely used mysql before), but I think it might show that there is UTF8 in the UTF8-collated table (although my terminal messes up the accented e during display). I then tried this: The output of the HEX(description) column for both of these commands is identical: Analysing the last part of that string, I think it is using C3 A9 for the accented 'e' in there: I only run mythconverg on this host. The configured mysql encodings are pretty much whatever was in the original mysql configuration file I copied from the internet somewhere (when I set up the mythtv system I didn't really know what I was doing mysql-wise; still don't for the most part). The only time I've fiddled with these settings was when I did the 0.21 to 0.22 upgrade which had latin8 data stored in utf8 tables (or something like that) and needed correction. However the above recording was created long after this upgrade. I do have an en_GB.UTF-8 locale defined on my system (along with an ISO-8859-1 one), although oddly my LANG is configured only as 'en_GB' so I'm not sure if my system is actually using UTF8 for terminals etc (which I would expect it to be really). I will look into fixing that, but will hold off until we've finished the diagnosis here unless otherwise instructed. |
|
Before going any further, are you in a position to try the beta? Your Android If not, then I'd try this (trying to prove where the conversion from tee /tmp/utf8.test
select description from recorded where title='New Girl' and subtitle ='Bachelorette Party';Then |
|
I've now tried the beta on the same Device (Nexus 10, Android 4.3) - that's quite a face lift. I still see no recordings, and "monitor" shows several occurrences of the same/a similar crash (except this time it appears before I directly tap the recordings tab; presumably the app retrieves the data on startup): I ran the test you suggested (although I used xxd not hd) and it looks like the character is being converted correctly: So marking that up a bit for ease For completeness I repeated this test after having executed 'set NAMES utf8' and verified that 'C3 A9' appeared. |
|
With the beta, I get a repeatable crash on exit if I do the following:
edit: Version number listed next to MAF is 1.20.0 - installed from play store about 10 minutes ago. |
|
1st, thanks for testing the beta. I was concerned because the developers are If I"m reading this correctly, in the DB, When you My backend starts like this: I haven't labeled this as a bug yet, because I don't know if JSON can ignore the And, yes, please open a new issue with the above in it, thanks again (I tried it on |
|
OK. I'll open another issue for that crash. Potentially that crash only occurs where my earlier crash has happened. I'll stick with the beta for the moment as it seems about as functional as the previous version for me :) I appreciate that there may not be a lot that the android front-end can do if it is sent invalid data. Of course ideally it would ignore that recording, display the rest and display a list of errors. As you say that may not be possible. What was getting me about this before was there wasn't an error - the application just stated "I've got all the recordings fine". From a usability perspective an error message would be nice. According to the mysql manual, set names tells the server that the character set the connected client is using. I'm not a mysql expert - I just pulled the command from the oreilly page. See http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html Even if this isn't eventually raised as a bug, I'm at least hoping I can reconfigure the system to fix this.
My backend doesn't appear to have any special settings/environment variables when executed, but I'm not massively sure on this. I'm wondering if mysql is converting the data just fine, and this is a mythbackend bug. Perhaps as part of the mythtv encoding to JSON format it assumes the data is already in UTF8 format and requires no conversion when it isn't (it might be latin8). Clearly the xml output code converts correctly. I don't know anything about the mythtv codeso wouldn't really know where to start looking. I also have no idea what connection settings the backend would use - it might explicitly request usage of UTF8 in all cases. |
|
I just added the é to one of my recordings and refreshed my screen. No failure. Note, the é displayed in both MythTV and in MAF A-OK. Does MythTV display |
|
Yes mythfrontend displays the accented e and the De Ja Vu title without problem. I'll start working on editing out the accented characters in the database and experimenting with my system configuration a little bit in a reversible way. |
|
I'd suggest using the frontend metadata editor, just 'cause it's safer. |
|
Good news: Removing the accented characters from the metadata (using the editor as you suggested) allowed MAF to display my recordings! I even managed to play back an episode! I then cleared MAF's cache and re-added the accented e to one of them - once again MAF failed to download any recordings (even after I hit refresh) - although I didn't check the android log to confirm why. Removing the character and hitting refresh again presented me with a populated recordings screen. I tried changing the mysql configuration to use all UFT8, but that doesn't seem to solve the problem - MAF still cannot access my recordings in this case. So I think my next step is to switch the backend (probably along with my entire environment) to using UTF8 and see if that allows proper use of multi-byte characters. |
|
@dmfrey, any way to absorb the bad data and prevent the crash? I can add a Wiki item about this, but the ideal would be if the app Good to know that we're trying to solve the correct problem. Thanks for Here's what I do (in part) to start my backend: But that's how Ubuntu does it. There was much discussion on the mythtv-users list /etc/defaults/local holds the proper settings on my machine, so sourcing it in the |
I added some catches in for the MythServicesApiRuntimeException. Unfortunately, i think the only thing we can do at this point is allow it to fail gracefully. If the incoming data isn't completed or unparsable in any way, I don't think it can partially proceed. These parsing exceptions are at too deep a level to allow the app to try to access any of the data and load it.
|
When I test this without setting LANG/LC_TYPE, the letter é simply doesn't |
|
@Speckto, 1.20.2 was released at about 1300UTC so it should |
|
@Speckto, I also tried setting my LANG/LC_TYPE to only en_GB and didn't Please retry with 1.20.2. I know you already tested 230. Thanks! |
|
So I retested this issue (without any character set reconfiguration) using the latest beta. I'm not likely to get around to testing with mythbackend working in different character sets until the weekend. I still don't see any recordings (which from Dan's note is expected) - I understand that there isn't anything you can do to fix the parsing. That makes sense. What happens: I was surprised to see that there is still no error message displayed in the app - it fails silently. The log file shows that a MythServicesApiRuntimeException is being thrown now (see below). Is this the expected behaviour with the beta or were you expecting to see some sort of error message? The new crash in case you are interested: |
|
@Speckto, thanks for the input. Looking back over these notes, you mentioned At your convenience, I'd appreciate it if you would try this modified No need to examine the JsonData.gz, but I'd like to know if the header in the response |
|
Doesn't look like it includes the charset unfortunately. |
|
I appreciate your checking. Plan C, if changing LANG/LC_TYPE works would be to modify a @dmfrey, the patch hasn't gotten any attention, so I'm not hopeful. Plus, at the very best, |
|
we are now catching the mythtv api runtime exceptions. Form that object we On Thu, Oct 17, 2013 at 3:26 PM, billmeek [email protected] wrote:
My Google Profile http://www.google.com/profiles/dmfrey |
|
Finally! Got my system to fail. In order to see the blank recordings page, The 1st two lines of the error in LogCat are: @dmfrey, could the 2nd line of the above be checked for "Invalid UTF-8" and @Speckto, I'm comfortable saying this is a backend configuration issue. But MAF If you get a chance, I think you can solve the problem by just starting the backend with: Thanks for hanging in there with us. |
|
@dmfrey The new code is in the catch {} about half way down... public boolean process( final Context context, final LocationProfile locationProfile ) {
Log.v( TAG, "process : enter" );
if( !NetworkHelper.getInstance().isMasterBackendConnected( context, locationProfile ) ) {
Log.w( TAG, "process : Master Backend '" + locationProfile.getHostname() + "' is unreachable" );
return false;
}
mMythServicesTemplate = (MythServicesTemplate) MythAccessFactory.getServiceTemplateApiByVersion( mApiVersion, locationProfile.getUrl() );
if( null == mMythServicesTemplate ) {
Log.w( TAG, "process : Master Backend '" + locationProfile.getHostname() + "' is unreachable" );
return false;
}
boolean passed = true;
try {
downloadRecorded( context, locationProfile );
} catch( Exception e ) {
Log.e( TAG, "process : error", e );
if( e.toString().contains( "Invalid UTF-8" ) ) {
// IF THIS WORKS, ADD REAL STRINGS!!!
Log.e( TAG, "INVALID UTF-8 DETECTED!!! Try to open a Progress Dialog, context='" + context + "'" );
mProgressDialog = new ProgressDialog( context );
mProgressDialog.setTitle( context.getResources().getString( R.string.gallery_sg_exist_create ) );
mProgressDialog.setMessage( context.getResources().getString( R.string.gallery_sg_exist_create ) );
mProgressDialog.setCancelable(false);
mProgressDialog.setButton(DialogInterface.BUTTON_NEGATIVE, context.getResources().getString( R.string.gallery_sg_exist_create ), new DialogInterface.OnClickListener() {
@Override
public void onClick(DialogInterface dialog, int which) {
dialog.dismiss();
}
});
mProgressDialog.show();
}
passed = false;
}
Log.v( TAG, "process : exit" );
return passed;
} |
|
I can confirm that changing the init.d script for mythbackend to explicitly set the LANG and LC_CTYPE to en_GB.UTF8 resolves the issue and the recordings work even with accented characters. Switching my backend environment to use UTF8 primarily also resolves this issue for me. I entirely agree that this is a backend configuration issue. This kind of thing is one of the risks of configuring a distro like Gentoo without knowing what you are doing - but that was kind of the point originally: to learn how Linux slots together :) Thank-you for your assistance with tracking down the cause of this! Having MAF display an error when this occurs would be extremely useful in case someone else hits this issue. When do you usually close issues like this? If so, I'll leave it open until that has occurred unless you want me to do otherwise. |
|
Am I remembering this correctly @Speckto, but did you previously say that I am wondering if this was just left over from old configuration. I On Sat, Oct 19, 2013 at 8:41 AM, Speckto [email protected] wrote:
My Google Profile http://www.google.com/profiles/dmfrey |
|
Yes it was upgraded from 0.21. (possibly an earlier version than that as well). I don't think this is left-over configuration, but bad initial configuration on my part. I always intended to use UTF8 system wide from the start - I'd even added the locale - but I never actually switched to using it (mistake). Perhaps mythbackend eventually started to require use of UTF8 sometime after 0.21 but I missed that upgrade step if it did. |
|
Ah, gotcha. I thought maybe this would have been in the myth scripts @billmeek we should come up with a workflow to add into the app to alert On Sat, Oct 19, 2013 at 8:53 AM, Speckto [email protected] wrote:
My Google Profile http://www.google.com/profiles/dmfrey |
|
@Speckto, thanks for testing the configuration on your BE. Normally, I like the @dmfrey, agreed, I'll respond off-list with some ideas. |
Hi,
I am using version 1.10.1 of the android app.
I run mythtv 0.26.0.
When I access the recordings screen, no recordings are displayed. The program guide does download ok, so it appears to have connectivity to the backend.
The application starts and appears to sync the program guide. I tap the recordings icon and I see a message "Recordings are up to date!" but no recordings are displayed.
I enabled logging on mythbackend (the upnp stuff from the wiki) and connected "monitor" up to my android device then captured the logging from when I clicked the 'recordings' icon:
Backend log shows that the connection is being made and urls are being requested:
I had a look at the log on the android device, and after clicking the recordings button, I see an exception being thrown which looks like it may be related:
I used wget to access the Dvr/GetRecordedList xml, and iconv seems to be able to convert the retrieved file from UTF-8 without any issues.
There is in fact a repeating error whilst downloading the program guide data as well - not sure if it is related:
The program guide appears to function correctly - but I haven't looked at it in detail.
Any idea how I can get my recordings screen working?
Thanks
The text was updated successfully, but these errors were encountered: