Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I cannot delete the language I downloaded #792

Closed
suatsuphi opened this issue Aug 21, 2023 · 23 comments
Closed

I cannot delete the language I downloaded #792

suatsuphi opened this issue Aug 21, 2023 · 23 comments
Assignees
Milestone

Comments

@suatsuphi
Copy link
Contributor

Hi,

It gives me an error when I delete the language I downloaded from Available Languages in the Languages section.

I can delete the language I added manually.

https://ibb.co/qpnBGgB

@suatsuphi suatsuphi added the bug label Aug 21, 2023
@muratcakir muratcakir added this to the 5.1.0 milestone Aug 21, 2023
@mgesing
Copy link
Contributor

mgesing commented Aug 21, 2023

I cannot reproduce this problem. Sqlite.Exception "database disk image is malformed" sounds like a corrupted database file.

@mgesing mgesing closed this as completed Aug 21, 2023
@suatsuphi
Copy link
Contributor Author

If I run the shrink (admin/maintenance/systeminfo/) , I can delete it.
var sql = $"VACUUM;PRAGMA wal_checkpoint=TRUNCATE;PRAGMA optimize;PRAGMA wal_autocheckpoint;";

if I can delete the language I added manually. this situation does not occur...

@mgesing mgesing added help wanted and removed bug labels Aug 22, 2023
@muratcakir
Copy link
Contributor

muratcakir commented Aug 22, 2023

@mgesing Please check again with SQLite. Could be a file locking problem caused by the fire-forget async operation.

@muratcakir muratcakir reopened this Aug 22, 2023
@mgesing
Copy link
Contributor

mgesing commented Aug 23, 2023

I still cannot reproduce it. Of course I have tried it with SQLite.

@mgesing mgesing closed this as completed Aug 23, 2023
@mgesing
Copy link
Contributor

mgesing commented Aug 23, 2023

Does this happen only with a specific language or with all downloaded languages?

@suatsuphi
Copy link
Contributor Author

suatsuphi commented Aug 23, 2023

yes, I all downloaded languages.

  • I downloaded the project from github.
  • I ran the project.
  • I did the installation in English (sqlite)
  • I downloaded Turkish from admin/language/list/.
  • I made the default language and the language of the admin panel Turkish. The language of the admin panel must be Turkish.
    https://ibb.co/vPjR8tM
  • I downloaded French from admin/language/list/.
  • Then it gives me an error when I try to delete the French I downloaded.

note: as I understand it that when I use the admin panel in Turkish, it does not allow me to delete the languages which I downloaded? allows me to delete languages that I have manually saved (admin/language/create/).

=================

I just realized now that this problem does not occur if shrink is run once. Now I download and delete and no error occurs.

I made Turkish the second language and then updated it with the refresh button. then I made it the first language again and made the admin panel language Turkish. when I add a new language by download, this error occurs again

@mgesing mgesing reopened this Aug 24, 2023
@mgesing mgesing added bug and removed help wanted labels Aug 24, 2023
@mgesing mgesing reopened this Aug 25, 2023
mgesing added a commit that referenced this issue Aug 25, 2023
@mgesing
Copy link
Contributor

mgesing commented Aug 28, 2023

Looks like the reason is a corrupted index on LocaleStringResource.

PRAGMA integrity_check;

gave me a hint about it.

PRAGMA writable_schema=ON;
VACUUM;

fixed it for me. (It cannot be excuted via backend, only via SQLite Db Browser.) Had to switch back from Turkish to English to get the language deleted afterwards. How can a database system be so fragile....

@suatsuphi
Copy link
Contributor Author

suatsuphi commented Aug 29, 2023

https://ibb.co/Q8tRBqk
You cannot delete a language you downloaded in screen language when you change the language. for example you downloaded a language in English screen language. if you switch to German or French, you get this index error

@suatsuphi
Copy link
Contributor Author

suatsuphi commented Aug 29, 2023

I figured it out... problem is here >> //var xmlResourceManager = scope.Resolve<IXmlResourceManager>();

I removed the DownloadCore method from being static - to be able to use the _xmlResourceManager from ctor

now it is working...


        private async Task DownloadCore(ILifetimeScope scope, LanguageDownloadContext context, CancellationToken cancelToken)
        {
            var appContext = scope.Resolve<IApplicationContext>();
            var asyncState = scope.Resolve<IAsyncState>();
            var httpClientFactory = scope.Resolve<IHttpClientFactory>();
            //var xmlResourceManager = scope.Resolve<IXmlResourceManager>();
            var db = scope.Resolve<SmartDbContext>();

            try
            {
                // 1. Download resources.
                var state = new LanguageDownloadState
                {
                    Id = context.SetId,
                    ProgressMessage = context.StringResources["Admin.Configuration.Languages.DownloadingResources"]
                };

                await asyncState.CreateAsync(state, null, false, CancellationTokenSource.CreateLinkedTokenSource(cancelToken));

                var client = httpClientFactory.CreateClient();
                var resources = context.AvailableResources.Resources.First(x => x.Id == context.SetId);
                var xmlDoc = await DownloadAvailableResources(client, resources.DownloadUrl, context.StoreUrl, cancelToken);

                if (!cancelToken.IsCancellationRequested)
                {
                    using var dbScope = new DbContextScope(db, minHookImportance: HookImportance.Essential);
                    await asyncState.UpdateAsync<LanguageDownloadState>(state => state.ProgressMessage = context.StringResources["Admin.Configuration.Languages.ImportResources"]);

                    var cultureCode = resources.Language.Culture;

                    // 2. Create language entity (if required).
                    var language = await db.Languages
                        .Where(x => x.LanguageCulture == cultureCode)
                        .FirstOrDefaultAsync(cancelToken);

                    if (language == null)
                    {
                        var maxDisplayOrder = await db.Languages
                            .Where(x => x.Published)
                            .MaxAsync(x => (int?)x.DisplayOrder, cancelToken);

                        language = new Language
                        {
                            LanguageCulture = cultureCode,
                            UniqueSeoCode = resources.Language.TwoLetterIsoCode,
                            Name = GetCultureDisplayName(cultureCode) ?? resources.Name,
                            FlagImageFileName = GetFlagFileName(cultureCode, appContext),
                            Rtl = resources.Language.Rtl,
                            Published = false,
                            DisplayOrder = maxDisplayOrder.HasValue ? maxDisplayOrder.Value + 1 : 0
                        };

                        db.Languages.Add(language);
                        await dbScope.CommitAsync(cancelToken);
                    }

                    // 3. Import resources.
                     await _xmlResourceManager.ImportResourcesFromXmlAsync(language, xmlDoc);

                    // 4. Save import info.
                    var serializedImportInfo = JsonConvert.SerializeObject(new LastResourcesImportInfo
                    {
                        TranslatedPercentage = resources.TranslatedPercentage,
                        ImportedOn = DateTime.UtcNow
                    });

                    var attribute = await db.GenericAttributes.FirstOrDefaultAsync(x =>
                        x.Key == "LastResourcesImportInfo" &&
                        x.KeyGroup == nameof(Language) &&
                        x.EntityId == language.Id &&
                        x.StoreId == 0, cancelToken);

                    if (attribute == null)
                    {
                        db.GenericAttributes.Add(new()
                        {
                            Key = "LastResourcesImportInfo",
                            KeyGroup = nameof(Language),
                            EntityId = language.Id,
                            StoreId = 0,
                            Value = serializedImportInfo
                        });
                    }
                    else
                    {
                        attribute.Value = serializedImportInfo;
                    }

                    await dbScope.CommitAsync(cancelToken);
                }
            }
            catch (Exception ex)
            {
                scope.Resolve<ILogger>().ErrorsAll(ex);
            }
            finally
            {
                if (asyncState.Contains<LanguageDownloadState>())
                {
                    asyncState.Remove<LanguageDownloadState>();
                }
            }
        }

@muratcakir
Copy link
Contributor

I removed the DownloadCore method from being static - to be able to use the _xmlResourceManager from ctor

But now you will have to deal with possible memory leaks, because DownloadCore is not awaited (fire-forget). The request ends before the method completes execution. The controller needs to be GCed, but DownloadCore is still using resources from its outer scope (the controller).

Better we investigate what's going on in IXmlResourceManager.

@mgesing
Copy link
Contributor

mgesing commented Aug 30, 2023

@suatsuphi Please test if the problem was fixed by b9110c5 for you, too.

@suatsuphi
Copy link
Contributor Author

suatsuphi commented Aug 30, 2023

it is working now with the patch that you made.

The index problem occurs when downloading, not deleting. The reason for this seems to be xmlResourceManager. Calling from xml ctor as below solves the problem. No index problem occurs as below.

https://ibb.co/LYMhm4M

[Permission(Permissions.Configuration.Language.EditResource)]
public async Task<IActionResult> Download(int setId)
{
    var resources = await CheckAvailableResources();

    if (resources.Resources.Count > 0)
    {
        var ctx = new LanguageDownloadContext(setId)
        {
            AvailableResources = resources,
            StoreUrl = Services.StoreContext.CurrentStore.GetBaseUrl(),
            StringResources = new()
            {
                { "Admin.Configuration.Languages.ImportResources", T("Admin.Configuration.Languages.ImportResources") },
                { "Admin.Configuration.Languages.DownloadingResources", T("Admin.Configuration.Languages.DownloadingResources") }
            }
        };

        _ = _asyncRunner.RunTask((scope, ct, state) => DownloadCore(_xmlResourceManager, scope, state as LanguageDownloadContext, ct), ctx);
    }

    return RedirectToAction(nameof(List));
}

private static async Task DownloadCore(IXmlResourceManager xmlResourceManager,ILifetimeScope scope, LanguageDownloadContext context, CancellationToken cancelToken)
{
    var appContext = scope.Resolve<IApplicationContext>();
    var asyncState = scope.Resolve<IAsyncState>();
    var httpClientFactory = scope.Resolve<IHttpClientFactory>();
    //var xmlResourceManager = scope.Resolve<IXmlResourceManager>();
    var db = scope.Resolve<SmartDbContext>();
...

XmlResourceManager.cs : I have culture in XmlResourceManager - even if I make the value here in English the same problem occurs. Like the exact solution above...

public partial class XmlResourceManager : IXmlResourceManager
{
    [GeneratedRegex("^resources.(.+?).xml$", RegexOptions.IgnoreCase | RegexOptions.Compiled, "de-DE")]
    private static partial Regex FileNameRegEx();

@mgesing
Copy link
Contributor

mgesing commented Aug 30, 2023

You do not understand scoping. Your code does not insert any string resource. That's why the database index does not get corrupted and why there is no "database disk image is malformed".

@suatsuphi
Copy link
Contributor Author

you are right...

but I think the problem is the XmlResourceManager.
Could I use CurrentCulture in the GeneratedRegex structure?

https://ibb.co/FmhPQny

@mgesing
Copy link
Contributor

mgesing commented Aug 31, 2023

I changed this 271a052. This is Intellisense suggested and auto generated code from yesterday commit. If its German or invariant culture should not make any difference in this particular case. I'm surprised that bothers you.

@suatsuphi
Copy link
Contributor Author

I thought the problem was with culture. I think I found out what the problem was. I tested it locally and it works fine.
https://ibb.co/3dHvygm

// 3. Import resources. 
//await xmlResourceManager.ImportResourcesFromXmlAsync(language, xmlDoc);
// for test
var xmlDoc2 = XDocument.Load(@"C:\Users\FB\Downloads\smartstore-string-resources-5.0.5.0-de-de.xml");
await xmlResourceManager.ImportResourcesFromXmlAsync(language, xmlDoc2.ToString());

I need to fix two ResourceValue values in all languages so that I can test them by downloading. otherwise it causes indexing and then malformed error.

	<LocaleResource Name="admin.system.systeminfo.appversion">
		<Value>Smartstore version</Value>
	</LocaleResource>
	<LocaleResource Name="admin.system.systeminfo.appversion.hint">
		<Value>Smartstore version</Value>
	</LocaleResource>

I think the problem will be solved when we fix the "info" here in all languages. Because the "i" character is not equal to the "I" character.

Can you fix this problem in xml files?

@mgesing
Copy link
Contributor

mgesing commented Sep 2, 2023

Interesting, but this looks like a workaround. There are very many resource names that are lowercase and contain an i. Should they all be changed manually, for all 32 languages?

@suatsuphi
Copy link
Contributor Author

suatsuphi commented Sep 2, 2023

No, just this tag > systeminfo, because the use is different
just two record (LocaleResource ) all language record...
the problem arose because these two records were written in lower case. this is enough systemInfo
https://ibb.co/L9tYkdD

like

	<LocaleResource Name="Admin.System.SystemInfo.Appversion">
		<Value>Smartstore version</Value>
	</LocaleResource>
	<LocaleResource Name="Admin.System.SystemInfo.Appversion.Hint">
		<Value>Smartstore version</Value>
	</LocaleResource>

@mgesing
Copy link
Contributor

mgesing commented Sep 2, 2023

Why these two? Why is the usage different? The XML contains a lot of names in lower case and LocalizedDisplay is always in pascal case. So the case applies to many string resources.

@suatsuphi
Copy link
Contributor Author

Why these two?

because these two records are not pascal case

The XML contains a lot of names in lower case and LocalizedDisplay is always in pascal case.

where is the converter that LocalizedDisplay is always in pascal case? Maybe something can be done here?

So the case applies to many string resources.

No, only pascal case applies to unused 'LocaleResource - Name' and just thesee two records
admin.system.systeminfo.appversion
admin.system.systeminfo.appversion.hint
All other entries are written within Pascal Case rules. I checked them.

We had a similar problem here.
#714
It was fixed when the related records were made pascal case.

mgesing added a commit that referenced this issue Sep 4, 2023
@mgesing
Copy link
Contributor

mgesing commented Sep 4, 2023

There are more than the two resources in lower case. These are very old resources that have been in the XML file from the beginning, not added later. I have fixed Admin.System.SystemInfo.AppVersion via migration now. But this is just the beginning. When you download Turkish you still get the lower case version though. I don't want to change it manually in Smartstore Translate. But you will get it after the next published Smartstore version when it will be changed there too. Smartstore Translate routinely updates the resources a short time after a new Smartstore version has been published.

@suatsuphi
Copy link
Contributor Author

thank you

I looked at the records and there are only 14 records with lowercase typed - pascal case does not fit.

account.login.newcustomertext
admin.appnews
admin.configuration.deliverytime.added
admin.configuration.deliverytime.deleted
admin.configuration.plugins.description.step1
admin.configuration.themes.option.defaultdesktoptheme.hint
admin.configuration.themes.option.defaultmobiletheme.hint
admin.orders.pdfinvoice
admin.pagetitle
admin.system.systeminfo.appversion
admin.system.systeminfo.appversion.hint
admin.system.warnings.incompatibleplugin
order.getpdfinvoice
products.callforprice

only two of these records are causing problems.

admin.system.systeminfo.appversion
admin.system.systeminfo.appversion.hint

the reason why these are causing problems is that the pascal case of the previous admin.system.systeminfo info is set differently

Admin.System.SystemInfo
Admin.System.SystemInfo
Admin.System.SystemInfo.AppDate
Admin.System.SystemInfo.AppDate
Admin.System.SystemInfo.AppDate.Hint
Admin.System.SystemInfo.AppDate.Hint
Admin.System.SystemInfo.ASPNETInfo
Admin.System.SystemInfo.ASPNETInfo
Admin.System.SystemInfo.ASPNETInfo.Hint
Admin.System.SystemInfo.ASPNETInfo.Hint
Admin.System.SystemInfo.DatabaseSize
Admin.System.SystemInfo.DatabaseSize
Admin.System.SystemInfo.DatabaseSize.Hint
Admin.System.SystemInfo.DatabaseSize.Hint
Admin.System.SystemInfo.DataProviderFriendlyName
Admin.System.SystemInfo.DataProviderFriendlyName
Admin.System.SystemInfo.DataProviderFriendlyName.Hint
Admin.System.SystemInfo.DataProviderFriendlyName.Hint
Admin.System.SystemInfo.DbTableInfo
Admin.System.SystemInfo.DbTableInfo
Admin.System.SystemInfo.DbTableInfo.NumRows
Admin.System.SystemInfo.DbTableInfo.NumRows
Admin.System.SystemInfo.DbTableInfo.TableName
Admin.System.SystemInfo.DbTableInfo.TableName
Admin.System.SystemInfo.DbTableInfo.TotalSpace
Admin.System.SystemInfo.DbTableInfo.TotalSpace
Admin.System.SystemInfo.DbTableInfo.UnusedSpace
Admin.System.SystemInfo.DbTableInfo.UnusedSpace
Admin.System.SystemInfo.DbTableInfo.UsedSpace
Admin.System.SystemInfo.DbTableInfo.UsedSpace
Admin.System.SystemInfo.GarbageCollect
Admin.System.SystemInfo.GarbageCollect
Admin.System.SystemInfo.GarbageCollectSuccessful
Admin.System.SystemInfo.GarbageCollectSuccessful
Admin.System.SystemInfo.HTTPHOST
Admin.System.SystemInfo.HTTPHOST
Admin.System.SystemInfo.HTTPHOST.Hint
Admin.System.SystemInfo.HTTPHOST.Hint
Admin.System.SystemInfo.IsFullTrust
Admin.System.SystemInfo.IsFullTrust
Admin.System.SystemInfo.IsFullTrust.Hint
Admin.System.SystemInfo.IsFullTrust.Hint
Admin.System.SystemInfo.LoadedAssemblies
Admin.System.SystemInfo.LoadedAssemblies
Admin.System.SystemInfo.LoadedAssemblies.Hint
Admin.System.SystemInfo.LoadedAssemblies.Hint
Admin.System.SystemInfo.OperatingSystem
Admin.System.SystemInfo.OperatingSystem
Admin.System.SystemInfo.OperatingSystem.Hint
Admin.System.SystemInfo.OperatingSystem.Hint
Admin.System.SystemInfo.ServerLocalTime
Admin.System.SystemInfo.ServerLocalTime
Admin.System.SystemInfo.ServerLocalTime.Hint
Admin.System.SystemInfo.ServerLocalTime.Hint
Admin.System.SystemInfo.ServerTimeZone
Admin.System.SystemInfo.ServerTimeZone
Admin.System.SystemInfo.ServerTimeZone.Hint
Admin.System.SystemInfo.ServerTimeZone.Hint
Admin.System.SystemInfo.UsedMemorySize
Admin.System.SystemInfo.UsedMemorySize
Admin.System.SystemInfo.UTCTime
Admin.System.SystemInfo.UTCTime
Admin.System.SystemInfo.UTCTime.Hint
Admin.System.SystemInfo.UTCTime.Hint
admin.system.systeminfo.appversion
admin.system.systeminfo.appversion.hint

Admin.System.SystemInfo > where the letters S and I are capitalized

but for the appversion resourcename it stays as Systeminfo. So it creates a different directory and this creates a problem. I don't understand at what stage this occurs, but I am sure it is related to the cache because I manually delete the relevant records and if I don't delete the cache, it gives an error. So I have to delete the cache. There is a situation related to the pascal case between the cache and the database beacuse SystemInfo is not equal Systeminfo

@mgesing
Copy link
Contributor

mgesing commented Sep 4, 2023

admin.system.systeminfo.appversion has been changed. admin.system.systeminfo.appversion.hint no longer exists (because it has the same value). So it is fixed now. Despite note on Smartstore Translate. And let symptoms be symptoms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants