DSM 7 support framework design

[ymartin59]

I propose to collaboratively edit this issue description (with comments for discussions about points) so that to prepare how framework has to evolve.

Objectives:

• if possible, package should run seamlessly on both DSM 6 and DSM 7
  • this is not possible, packages for DSM 6 and DSM 7 are not compatible
  • packages for DSM 7 need at least os_min_ver="7.0-40000"
• reduce as much as possible package maintenance: implement support in framework make files and generated/embedded scripts

Design proposals:

• ✔️ At upgrade "var" content is restored as "home" in both DSM 6 and DSM 7
• Migrate to the new resource/permission file/specification which help implement common uses cases that otherwise required root Privileges (may not be required see comment, but is something we are working towards)
  • ✔️ SPK_COMMANDS and SPK_LINKS have to generate resource file for links
  • ✔️ installer has to generate database resource file from wizard for concerned application
  • ❌ Use privilege file to add package to a group e.g. sc-media
  • ✔️ Use privilege file to add create shared folder

To investigate

• Privileges: what should be done?
• ❌ Verify that adding package to a group e.g. sc-media still works from the install script.

Minimal requirements for DSM7

• os_min_ver= must be present and the value at least "7.0-40000"
• A conf/privilege file must exist (and have "defaults": { "run-as": "package" } [other values not validated yet])

[publicarray]

I've played around a little in my own fork, just to see what breaks and I think I got the minimal requirements done so packages can be "installed" but probably won't work. And binary packages can now be linked to /usr/local using Synology's resource file.

[th0ma7]

OPTIONAL: There are a few changes I'd like to complete on the framework before jumping on the dsm7 bang-wagon:

1. Review the DSM vs SRM management by using the new TC_TYPE variable now in place in the toolchain
2. Review toolchains naming to be singular and use long naming for mk/spksrc.tc*.mk for mk/spksrc.toolchain*.mk
3. Review the ARM7 exception used for some packages such as with go dependencies in order to have a a generic approach such as ARCH_NO_DUPE or FORCE_ARCHS (help on naming appreciated) pointing towards default available arches allowing to remove dupes for other arches as well as needed and also allow forcing specific sub-arches. This would also allow eliminating newly added syno-apollolake-6.2 in the all-supported builds.
4. Last but not least, a little more far-stretched, review PLIST management not following regular cookie building makefile layout.

And forgot to mention meson/ninja build support as many packages cannot be further updated anymore...

[hgy59]

synology has published a beta release of the Development documentation DSM_Developer_Guide_7_0_Beta.pdf.
It is newer than the preview version.

One interesting difference I found is for 3rdparty developers:

1. Force lower privilege for package

All packages should provide conf/privilege with package in run-as explicitly. Any privileged operation should be
accomplished via resource worker.
But if you are 3rdparty developer, you can still use root privilege.

[publicarray]

@ymartin59 Thanks, but why not link to the file? https://global.download.synology.com/download/Document/Software/DeveloperGuide/Firmware/DSM/7.0/enu/DSM_Developer_Guide_7_0_Beta.pdf
I don't know but with the current build, when I make a package that asks for root "run-as": "root" I get this:

Failed to install. The package should run with a lower privilege level. Please contact the package developer to modify the privilege settings.

[hgy59]

@publicarray thanks for the link (comment updated).
I think that this will work with DSM 7 beta (we are still on DSM 7 preview).

[publicarray]

That may be true. but I think sooner or later permissions will be more restricted over time, maybe at DSM version 24 ;)
Still think it's worth to migrate some of the shell scripts to their resource file equivalent. What do you think?

[hgy59]

Yes, agree. We should use the official kind of resource scripts whereever provided.

EDIT:
the following is proposed already

We should use resource scripts for creating links too.
As I introduced SPK_COMMANDS and SPK_LINKS for commandline tools, it would be worth to create the resource scripts based on these (or slightly adjusted) variables.

[publicarray]

resource scripts for creating links too.

Yeah I've worked on that in my dsm7 branch. You can see the result in the synocli-disk GitHub release here. https://github.com/publicarray/spksrc/releases

Agreed. Currently I assume SPK_ COMMANDS are binaries so for other resource types new variables need to be made. Using SPK_COMMANDS would seamlessly work on existing scripts.

I think I'll create PRs once Synology releases the beta and once I'm more confident that it works correctly.

[publicarray]

To maniuplate json I would like to jq as a dependecy to the docker image and spksrc. Why is sed not enough? Once arrays are used it makes code hard to read and error prone, jq ensures that we generate valid json.

e.g. To add a package to the sc-media and sc-example group in the privilege file:

$ jq --arg packagename sc-media '."join-pkg-groupnames" += [{$packagename}]' privilege | jq --arg packagename sc-example '."join-pkg-groupnames" += [{$packagename}]'
{
  "defaults": {
    "run-as": "package"
  },
  "join-pkg-groupnames": [
    {
      "packagename": "sc-media"
    },
    {
      "packagename": "sc-example"
    }
  ]
}

Edit: there are other ways to do this (see below) but the above is easy to use in a loop.

$ jq --arg pkg1 sc-media --arg pkg2 sc-example '."join-pkg-groupnames"+=[{packagename: $pkg1},{packagename: $pkg2}]' privilege
{
  "defaults": {
    "run-as": "package"
  },
  "join-pkg-groupnames": [
    {
      "packagename": "sc-media"
    },
    {
      "packagename": "sc-example"
    }
  ]
}

Edit2: Create the minium needed for a privilege file

$ echo '{}' | jq --arg run_as package '.defaults = {"run-as": $run_as}'
# or
$ echo '{}' | jq --arg key run-as --arg value package '.defaults[$key] = $value'

{
  "defaults": {
    "run-as": "package"
  }
}

I'm happy to work with alterntives too :) Just please don't make use sed for json manipulation 😆

[hgy59]

@publicarray may you please rebase the dsm7 branch to current head@master?
I would like to build some current packages (like ntopng) for my DS218+ that has installed DSM7 preview.

[publicarray]

@hgy59

Yup. It's Done 👍

[publicarray]

DSM7 is now in Beta https://www.synology.com/en-global/beta/DSM70Beta

[kalatos86]

Is there any information when the community packages might have update to be compatible with DSM 7?

[publicarray]

@kalatos86 Since this is an open source project where people donate their time to contribute I can't give you an ETA. But you are more than welcome to build packages / contribute on the dsm7 branch

[publicarray]

DSM7 and Logs

I propose to change logging in DSM7, if we do nothing some logs will be lost between, package upgrades and install because the target/var folder is not permanent storage. (not a huge issue) Using var aka ${SYNOPKG_PKGVAR}/${SYNOPKG_PKGNAME}.log" fixes the above problem but turns out that the $SYNOPKG_PKGVAR variable does not exist in every stage of the installation cycle thus attempting to write the log file in /${SYNOPKG_PKGNAME}.log which fails in those stages. I think writing to stderr fixes these issues because now Synology is responsible for logging and log rotation (which it does on package reinstall) It also frees up some code in our scripts. Please let me know what you think. @ymartin59 @hgy59 @th0ma7

# /var/package/{package}/target/var/{package.log}.log (Current, main branch)
LOGFILE="${SYNOPKG_PKGDEST}/var/${SYNOPKG_PKGNAME}.log"

# /var/package/{package}/var/{package.log}.log (permanent storage, is in dsm7 branch)
LOGFILE="${SYNOPKG_PKGVAR}/${SYNOPKG_PKGNAME}.log"

# /tmp/synopkgmgr.log-{package} # I no idea how/where to retrieve it, since the file disappears immediately after install.
LOGFILE= ${SYNOPKG_TEMP_LOGFILE}

# /var/log/packages/{package}.log (I think that this is the best option)
echo "$1" 1>&2 #  stderr

# for completeness: show the message to the user
echo "$1" # (during install only, /var/log/packages/{package}.log when service is running)

 # /var/log/messages
logger -p 3 "$1"

[th0ma7]

Sounds like a good idea but I haven't looked much on that part of the code, neither on the dsm7 update yet. @ymartin59 and @hgy59 might have more to say on this.

[publicarray]

Thanks @th0ma7

While I was looking through some code I found a few packages uses busybox to delete users/groups from what I assume a migration from around DSM5 to DSM6? (can someone confirm that that is the intent?) Since I expect it already hard to migrate settings from DSM6 I want to drop busybox completely in DSM7 and not support package migration from DSM5 to DSM7, (IMHO It doesn't make sense to do so anyway)

[publicarray]

@ymartin59 @hgy59 What arches should we build for DSM7.0? I think this should cover everything:

make -j`nproc` arch-x64-7.0 arch-armv7-7.0 arch-aarch64-7.0 arch-evansport-7.0

[publicarray]

@hgy59 I've added SPK_DEPENDS to avoid confusion: https://seby.io/download/synology-7.0/adminer_noarch-dsm7_4.7.6-3.spk
publicarray@3dfb3d4

Yea it works on my machine:tm: The new folder should have been /var/services/web_packages/adminer maybe the migration is broken.

If you continue to have problems can I suggest reinstalling web station?

Was there a reason packages still use Apache 2.2? I'm actually surprised Synology still ships it. From apache's website:

The Apache HTTP Server Project had long committed to provide maintenance releases of the 2.2.x flavor through June of 2017. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases.

FYI same for php 5.6, 7.0 and 7.2: https://www.php.net/supported-versions.php (7.4 is the only realistic version to support right now)

[hgy59]

You cannot uninstall apache 2.2 without accepting the following warning in DSM7:

Uninstalling this package will cause abnormalities in the following functions. 
Do you mant to continue?
Some of the Web Stations's services will not be able to function properly, including default
back-end, personal website, and Virtual Host.

Hopefully apache 2.2 (and other outdated packages) will be removed in the final DSM7...

[publicarray]

Does the warning show up even when you have both apache versions installed? I don't use those functions but I can confirm that the default server works with just apache2.4 installed.

[publicarray]

Oh in DSM6 the checkbox for personal website is disabled until I install old versions of php and apache. DSM7 allows you to mix and match php/apache versions

[hgy59]

Does the warning show up even when you have both apache versions installed? I don't use those functions but I can confirm that the default server works with just apache2.4 installed.

Yes, I have both versions (2.2 and 2.4 installed) and you can uninstall apache2.4 without warning.

[publicarray]

I think that warning only shows up if you have a 'personal website or Virtual Host' setup from DSM6. I accepted the warning once but it no longer shows up when I uninstall 2.2.

[publicarray]

What is configured for your default server?

[publicarray]

If I install 2.2 and set the default server to 2.2 I get the warning. If I set it to 2.4 and uninstall 2.4 I also get the warning. On Nginx I don't. I guess it's to prevent unexpected errors. I which the warning was clearer though.

To sum up just update the setting for your default server to 2.4 or nginx (unless you explicitly require 2.2 for some reason).

[hgy59]

Yes you have to remove/update all the Web Service Portal configurations and Script Language Settings referencing outdated apache and php versions. Then you can uninstall the appropriate packages without warnings.

[digitalbox94]

Another interesting source of information regarding DSM7 package : https://github.com/SynologyOpenSource/ExamplePackages

[publicarray]

Thanks @digitalbox94 but unfortunately it's no very useful. It only demonstrates the new /usr/local/bin/ linking. We already implemented it on the dsm7 branch.

[digitalbox94]

Ok I've played a little bit with the Icecast package and DSM7 and here are my findings :

• Cross compile is successful for braswell DSM 7
• Installation on DSM 7 is OK (no error raised)
• Running is not OK by default because the folder /usr/local/icecast is not created during installation (to be investigated)

By doing some manual tricks directly on the NAS (configuration files & directory creation) I've got Icecast server running fine on my DS716+II with an ezstream flow (ezstream is also embedded in this package).
It means it's a good starting point before having a package 100% DSM 7 compatible.

I will continue my investigation but I you have any comments/advice to share for this DSM 7 topic they are welcome.

[publicarray]

Thanks @DigitalBox98, fixed in 9dd89e3

Most packages have old assumptions on the file layout and permissions so most will need to be manually migrated.

[digitalbox94]

Many thanks, the start of the server now works without any manual fix on DSM 7 ! :)

Regarding the "service-setup" file, I have identified an issue to fix regarding the "ice cast.xml" CFG-FILE where the correct line is :
CFG_FILE="${SYNOPKG_PKGDEST}/var/icecast.xml"
It will help to have correct login/password for the admin part

[DigitalBox98]

I would like to test the below "proof-of-concept" for DSM7 (but also for DSM6) :

• Propose a basic ImageMagick admin page (HTML/PHP) to have image conversion available directly in DSM
• Protect pages via Syno AUTH webapi (working fine on DSM7)

I'm actually fine tuning the solution on DSM7 because sometime the admin page is not available via DSM7 (access to /var/services/web_packages/imagemagick), and sometime it works (need to investigate further).
Putting the HTML/PHP pages on another directory (like previous method on DSM6 ie /var/services/web/imagemagick resolve 100% the previous issue on DSM7.

If I would like to propose a PR for this for DSM7, but what is the best below :

• Send this PR to a branch "imagemagick" created from master branch ?
• Send this PR to a branch "imagemagick" created from DSM7 branch ?

Another point : actually I have created & tested it on DSM7 with a branch created from DSM7 branch
Does it make senses to propose it for DSM6 with a branch "imagemagick" created from master branch ? (ie would the framework will generate the correct version for DSM6)

Let me know if you are interested to test this (at least the Syno AUTH webapi can be useful for other packages).

[EngMarc]

As for the _GLIBCXX_USE_C99_MATH and al, shouldn't we add them by default in mk/meson|cmake| if TC_GCC = 7.3.0 to make things simpler?

@th0ma7 when this works for ffmpeg too, this is a good proposal.
@th0ma7 is the a new ffmpeg for DSM 7 yet?

[erdoukki]

But if you are 3rdparty developer, you can still use root privilege.

This part was removed from the PDF ?

synology has published a beta release of the Development documentation DSM_Developer_Guide_7_0_Beta.pdf. It is newer than the preview version.

One interesting difference I found is for 3rdparty developers:

1. Force lower privilege for package

All packages should provide conf/privilege with package in run-as explicitly. Any privileged operation should be
accomplished via resource worker.
But if you are 3rdparty developer, you can still use root privilege.