Separate manifest generation, bundle generation, and bundle deployment

[colearendt]

Is there a reason that rsconnect requires building manifest.json for each deploy?

I think it makes sense that if the user already had a manifest.json file (i.e. after downloading a .tar.gz bundle from Connect), then there should be no reason for them to reinstall packages in order to generate another one. In theory, they should be the same.

Adding a parameter which defaults to NULL but passes the output to rsconnect:::bundleApp and allows bypassing the manifest.json build should be backwards compatible and would make migrating content from one Connect server to another much easier.

Of course, I might be missing an obvious reason that this is not available as an option. Not sure if packrat/packrat.lock file needs to be included in this discussion or not. In the case that you have downloaded a .tar.gz from Connect, you should have that file ready to go as well.

[jmcphers]

There's no reason we can't do this. We've also had customers request that we save the manifest somewhere useful, where it would could be recycled using a method like the one you describe. (Today, the manifest exists only temporarily on the local machine.)

We can't re-use the packrat lockfile, though; the packrat lockfile and the manifest include snapshots which are different in subtle but important ways. This issue has the gory details:

#143

Can you write a little more about your use case? (i.e. if you're only just looking to use rsconnect to bridge content from one connect server to another, maybe we should just allow uploading the .tar.gz bundle wholesale?)

[colearendt]

So if we do have the manifest.json, do we need the packrat lockfile, as well? I guess my thought / question was whether we need both of them to bypass the packrat lockfile build. I.e. can we build a lockfile from the manifest.json? Must the user provide both?

With respect to the use case, I am thinking along the lines of what @slopp was talking about in #143 . Ultimately, the goal is determining the minimal set of files that are necessary to drive a deployment. If we can determine that set of files, then we just need:

• a way to utilize those files if already provided to accelerate the deployment process and require less about the environment being deployed from
• a way to build those files separate from the deployment process if the user is so inclined

This will help with strides towards the following use cases:

• deploying from a Connect bundle to migrate content (the idea of deploying the bundle itself is intriguing to me... although we would still need to untar on the server side to install dependencies. It seems cleaner to just require the files to be untar'd)
• storing a minimal set of files to allow publishing directly from github
• scripting installation of content

[scottmmjackson]

@jmcphers I definitely think fire-and-forget tarball deploy would be useful. In particular, having a canonical way to prepare content for deploy without actually deploying it, and deploying prepared content without having to prepare it, would simplify a lot of the design choices in a handful of stories in our Connect backlog.

@colearendt IIRC:

• Every piece of content has a manifest.json
• Every piece of non-static content has a packrat.lock (note that Tensorflow Models count as "static" here, kind of a gotcha)

It might be a good idea for this repo to define a specification for manifest.json if we're going to start talking about deploying repos with preexisting manifests. Especially if we're anticipating people are going to be homebrewing or third-partying tools.

[slopp]

Full write up of goals here: https://docs.google.com/document/d/1PpDa-oIRuZUT9mJGrbofI8jhQlO_sdVMRgNeS3XLi0I/edit?usp=sharing

[aronatkins]

Also note that the bundle includes full DESCRIPTION files for the dependent packages.
That code is here:

rsconnect/R/bundle.R

Lines 563 to 587 in 31aaf03

	# Copy all the DESCRIPTION files we're relying on into packrat/desc.
	# That directory will contain one file for each package, e.g.
	# packrat/desc/shiny will be the shiny package's DESCRIPTION.
	#
	# The server will use this to calculate package hashes. We don't want
	# to rely on hashes calculated by our version of packrat, because the
	# server may be running a different version.
	lockFilePath <- snapshotLockFile(bundleDir)
	descDir <- file.path(bundleDir, "packrat", "desc")
	tryCatch({
	dir.create(descDir)
	packages <- na.omit(read.dcf(lockFilePath)[,"Package"])
	lapply(packages, function(pkgName) {
	descFile <- system.file("DESCRIPTION", package = pkgName)
	if (!file.exists(descFile)) {
	stop("Couldn't find DESCRIPTION file for ", pkgName)
	}
	file.copy(descFile, file.path(descDir, pkgName))
	})
	}, error = function(e) {
	warning("Unable to package DESCRIPTION files: ", conditionMessage(e), call. = FALSE)
	if (dirExists(descDir)) {
	unlink(descDir, recursive = TRUE)
	}
	})

Related PR (see the linked issues):
#42

[jmcphers]

I've updated the issue title to better align with our prior conversations on this topic. I think we've agreed that there are three distinct operations we need to expose:

1. Given an application folder, generate a manifest.json file that includes the proposed set of files along with the content type, etc.
2. Given a manifest.json file and an application folder, generate a bundle of files that can be deployed.
3. Given a bundle of files, deploy them as an application to Connect.

Right now deployApp does all these things. The proposal is that we'd refactor deployApp into three separate, exposed functions which did each step, and keep the deployApp interface stable for IDE use (it would just serve as a wrapper that performed each step in sequence).

[johanneswaage]

Hello!
For using Rstudio connect with Team Foundation Server as primary devops-platform in-house, I'm also very interested in being able to split up the bundle/deploy flow. Any news on this?

[slopp]

@johanneswaage yes - this feature is complete and is scheduled for release with RSC 1.7.0 later this week. Please let us know if the new capabilities meet your needs, and we would be happy to talk through the process if any parts of the new documentation are unclear. (See the RStudio blog for the release announcement which will include links to relevant docs).

[johanneswaage]

Hi again. I see Connect 1.7 has been updated with programmatic deployment through API, but how about through the rsconnect package, as described in this thread? Thanks

[slopp]

@johanneswaage - the process would be:

• R user creates a manifest using rsconnect::writeManifest
• The manifest is committed to the repo alongside the code
• From here, your CI process uses the APIs to deploy the content from the Git repo

Examples of step 3 are included here: https://github.com/rstudio/connect-api-deploy-shiny

[johanneswaage]

Thanks, that is handy, but a future request would definitely be to have manifest/bundle/deploy in rsconnect as well.

[slopp]

Can you elaborate on your use case? What are you trying to do?

cc: @colearendt

[hadley]

I'm closing since this is old, and looks mostly resolved, and for the bits that arent, I think #471 will help.