Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preprocess assets as Django management command #5

Merged
merged 4 commits into from
Jun 3, 2013

Conversation

natehardison
Copy link
Contributor

Rather than directly invoke command-line Python (and Mako) from the
assets Rakefile, or call an external Python script, use a Django
management command to preprocess all asset template files.

An "asset template file" is defined as a static asset file with a
file extension indicating that it needs to be run through a template
engine prior to Sass/CoffeeScript compilation or packaging with other
assets. The preprocess_assets management command will look through
all of the files listed in the STATICFILES_DIRS, preprocessing each
as needed. Preprocessing strips off the special template file
extension, creating a new file in the process.

Currently, the only variable accessible in an asset template file is
the THEME_NAME, defined in the settings.

@ghost ghost assigned cpennington May 30, 2013
@natehardison
Copy link
Contributor Author

@cpennington here's the real fix for that old assets.rake issue. I ended up putting the command inside the mitxmako app, but it'd be easy to move elsewhere if you'd prefer.

# just hardcode the asset filenames.
preprocess_with_mako("lms/static/sass/application.scss.mako")
desc "Preprocess all templatized static asset files"
task :preprocess, [:system, :env] do |t, args|
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rake passes all args down through the chain of commands...I hadn't known that. Pretty slick.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And, as I found out, that's only if a task that depends on this one takes these args. Otherwise, the defaults are used.

@natehardison
Copy link
Contributor Author

@chrisndodge this is the real real fix for the assets.rake problem. If there's a good way to test it out with a real production env.json file, then let me know.

@chrisndodge
Copy link
Contributor

Hmmm. Seems like you need to rebase?

@natehardison
Copy link
Contributor Author

Ergh. I sent this in before Marco merged release back into master. Will do
in a bit.

On Thursday, May 30, 2013, chrisndodge wrote:

Hmmm. Seems like you need to rebase?


Reply to this email directly or view it on GitHubhttps://github.com/edx/edx-platform/pull/5#issuecomment-18716923
.

Nate

@natehardison
Copy link
Contributor Author

Okay, @chrisndodge @cpennington, rebased to master post-release merge. Should be good to review.

@chrisndodge
Copy link
Contributor

OK, builds are failing, but it's not related to your work.

Anyhow, PR looks OK otherwise codewise. I'll try deploying this to stage@edge for testing purposes.

@chrisndodge
Copy link
Contributor

Sorry deploy is failing. Seems like it is trying to open up a log file in the wrong place. Typically the log is in /mnt/logs/|cms, lms, ....| Perhaps you can coordinate with @e0d or @jarv

Here is the stack trace:

30531 09:43:33 ] : [stage-edge-edxapp-001.m.edx.org:22] out: (See full trace by running task with --trace)
[ 20130531 09:43:34 ] : [stage-edge-edxapp-001.m.edx.org:22] out: django-admin.py preprocess_assets --traceback --settings=lms.envs.dev --pythonpath=.
[ 20130531 09:43:34 ] : [stage-edge-edxapp-001.m.edx.org:22] out: xmodule_assets common/static/xmodule
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: fatal: No tags can describe '5dfeca786ae403c5cfe468bea0e1d895645ac88d'.
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: Try --always, or create some tags.
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: Traceback (most recent call last):
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: File "/opt/edx/bin/django-admin.py", line 5, in
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: management.execute_from_command_line()
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: File "/opt/edx/local/lib/python2.7/site-packages/django/core/management/init.py", line 443, in execute_from_command_line
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: utility.execute()
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: File "/opt/edx/local/lib/python2.7/site-packages/django/core/management/init.py", line 382, in execute
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: self.fetch_command(subcommand).run_from_argv(self.argv)
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: File "/opt/edx/local/lib/python2.7/site-packages/django/core/management/init.py", line 252, in fetch_command
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: app_name = get_commands()[subcommand]
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: File "/opt/edx/local/lib/python2.7/site-packages/django/core/management/init.py", line 101, in get_commands
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: apps = settings.INSTALLED_APPS
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: File "/opt/edx/local/lib/python2.7/site-packages/django/utils/functional.py", line 184, in inner
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: self._setup()
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: File "/opt/edx/local/lib/python2.7/site-packages/django/conf/init.py", line 42, in _setup
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: self._wrapped = Settings(settings_module)
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: File "/opt/edx/local/lib/python2.7/site-packages/django/conf/init.py", line 135, in init
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: logging_config_func(self.LOGGING)
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: File "/usr/lib/python2.7/logging/config.py", line 777, in dictConfig
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: dictConfigClass(config).configure()
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: File "/usr/lib/python2.7/logging/config.py", line 575, in configure
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: '%r: %s' % (name, e))
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: ValueError: Unable to configure handler 'local': [Errno 13] Permission denied: u'/opt/wwc/log/edx.log'
[ 20130531 09:43:35 ] : [stage-edge-edxapp-001.m.edx.org:22] out: asset preprocessing failed!
[ 20130531 09:43:36 ] : [stage-edge-edxapp-001.m.edx.org:22] out: xmodule_assets common/static/xmodule
[ 20130531 09:43:38 ] : [stage-edge-edxapp-001.m.edx.org:22] out: node_modules/.bin/coffee --compile */static
[ 20130531 09:43:43 ] : [stage-edge-edxapp-001.m.edx.org:22] out:

@cpennington
Copy link
Contributor

@chrisndodge when you ran on stage, did you pass in an environment or system? It defaults to lms and dev, which means it would use lms/envs/dev.py, but on staging/production servers we'd want to use lms/envs/aws.py. I think that would contribute to the error message you were seeing.

@cpennington
Copy link
Contributor

👍 shipit

@chrisndodge
Copy link
Contributor

@cpennington this is the fab command I used:

fab tags:group=edxapp,env=edge_staging git:edx-platform=d3e2b87cd7466fed5e8c637aa826447b5d9cc450

@natehardison
Copy link
Contributor Author

Could it be that I write to stdout?

On Friday, May 31, 2013, chrisndodge wrote:

@cpennington https://github.com/cpennington this is the fab command I
used:

fab tags:group=edxapp,env=edge_staging
git:edx-platform=d3e2b87cd7466fed5e8c637aa826447b5d9cc450


Reply to this email directly or view it on GitHubhttps://github.com/edx/edx-platform/pull/5#issuecomment-18753360
.

Nate

@natehardison
Copy link
Contributor Author

@chrisndodge any way for me to reproduce the error you're seeing?

@chrisndodge
Copy link
Contributor

Hmmm. I don't have an idea off the top of my head. Can you use this script
to deploy to Stanford's hosting environment?

On Fri, May 31, 2013 at 5:53 PM, Nate Hardison [email protected]:

@chrisndodge https://github.com/chrisndodge any way for me to reproduce
the error you're seeing?


Reply to this email directly or view it on GitHubhttps://github.com/edx/edx-platform/pull/5#issuecomment-18773666
.

@natehardison
Copy link
Contributor Author

@jbau could you take a look at @chrisndodge's comments when you get a chance? This change is blocking some of the theming work that @caesar2164 and I are doing.

@jbau
Copy link

jbau commented May 31, 2013

@natehardison and I are on this. It seems like the write to stdout is a potential culprit. We're looking for workaround now.

@natehardison
Copy link
Contributor Author

Okay, I think @jbau and I have figured out what was going on. The <system>:gather_assets:<env> task depends on the :assets task, and I was counting on the :assets task receiving the <system> and <env> arguments to then pass to the :assets:preprocess task. However, the <system>:gather_assets:<env> task has the <system> and <env> arguments as part of its name, not as true arguments, so they don't get passed to dependent tasks.

The fix is to rewrite the <system>:gather_assets:<env> task as a wrapper around a :gather_assets task that takes the appropriate args. @cpennington, how's that sound? Would you prefer a different architecture there?

We also just found another bug, namely that I needed to add mitxmako as an INSTALLED_APP in the CMS as well.

@@ -319,6 +319,7 @@
'track',

# For asset pipelining
'mitxmako',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A deploy runs both lms:gather_assets:aws and cms:gather_assets:aws, so the CMS also needs access to the mitxmako app for the preprocess_assets management command.

@natehardison
Copy link
Contributor Author

Okay, @chrisndodge @cpennington, ready for review. @jbau verified that it deploys cleanly to our machines.

@chrisndodge
Copy link
Contributor

@natehardison Ok, was able to get around the problem with my 'sysadmin' repo (reverted back a few commits) and I've been able to push your branch out to stage@edge.

@cpennington do you want to do a final signoff on the code side of things?

Otherwise I'm +1

@natehardison
Copy link
Contributor Author

Failing because of pep8 and pylint violations. Will rebase.

@cpennington @ormsbee any thoughts on the code? In particular, thoughts on the assets.rake changes?

@chrisndodge
Copy link
Contributor

@natehardison can we finish off this PR on Monday? Seems like Cale already +1'd it up above.

@natehardison
Copy link
Contributor Author

I'll merge it today if that's okay.

On Sunday, June 2, 2013, chrisndodge wrote:

@natehardison https://github.com/natehardison can we finish off this PR
on Monday? Seems like Cale already +1'd it up above.


Reply to this email directly or view it on GitHubhttps://github.com/edx/edx-platform/pull/5#issuecomment-18812891
.

Nate

@chrisndodge
Copy link
Contributor

OK, I see we don't have a green checkmark on this PR. It's probably due to the fact Jenkins was broken late last week. Can you check in a small change (e.g. may a comment or a docstring) to kick off another build. Thanks.

Then when you see a green check, you can do the merge. Then let me know when it's done and I'll try a deploy to staging to verify it.

doctoryes pushed a commit that referenced this pull request Sep 15, 2014
doctoryes pushed a commit that referenced this pull request Sep 23, 2014
chrisrossi referenced this pull request in jazkarta/edx-platform Sep 24, 2014
You can now specify an arbitrary URL to a browser compatible video file
for the course intro video.
chrisrossi referenced this pull request in jazkarta/edx-platform Sep 26, 2014
Allow arbitrary embed code for intro video, in addition to direct URL
and YouTube video ID.
downzer0 pushed a commit that referenced this pull request Nov 17, 2014
downzer0 pushed a commit that referenced this pull request Nov 18, 2014
downzer0 pushed a commit that referenced this pull request Nov 21, 2014
dsego added a commit that referenced this pull request Apr 17, 2015
dsego added a commit that referenced this pull request Apr 18, 2015
dsego added a commit that referenced this pull request Apr 22, 2015
mbifulco referenced this pull request in gymnasium/edx-platform Aug 27, 2015
…orum

changing "discussion" tab to say "Forum"
singhularity pushed a commit to singhularity/edx-platform that referenced this pull request Sep 8, 2015
…OAuth to amplify-master

* commit 'a5d34f8bc42adddf91feddf650d8fa654d65b63d':
  Change http to https in redirect_uri
  Change the variable name for key and secret
  change amplify oauth provider from fqa to prod
  Fix pylint errors and refine the code
  change amplify oauth class to fix the link and redirect_uri problems
  delete unused library import
  Change amplify provider's name and change auth urls to prod
  import amplify profider from edx third party lib instead of python lib
  Add amplify as third party auth provider for edX/Amplify SSO
Bound3R referenced this pull request in eduNEXT/edx-platform Dec 20, 2016
naeem91 pushed a commit to naeem91/edx-platform that referenced this pull request Feb 5, 2017
removed correct/incorrect indicators
shrlyhe pushed a commit that referenced this pull request Jun 5, 2017
# This is the 1st commit message:
ENT-385 change error msg for confirm email

# This is the commit message #2:

remove changed msgid from po files

# This is the commit message #3:

add name to AUTHORS file

# This is the commit message #4:

remove lastfailed file

# This is the commit message #5:

GradingPolicyChanged Signal Handler

https://openedx.atlassian.net/browse/EDUCATOR-393

# This is the commit message #6:

EDUCATOR-435 | Include enrollment status in in course and problem grade reports.

# This is the commit message #7:

Change visibility to access.

EDUCATOR-396

# This is the commit message #8:

LEARNER-923 Make Pattern Library support tabs transparent

# This is the commit message #9:

Conform to WCAG 2 AA contrast minimums for Google OAuth.

Also apply variables for FB/Linkedin OAuth2 rather than
leave color hashes hanging around.

# This is the commit message #10:

Updated auto_auth endpoint to always return JSON

Defaulting to a plaintext response makes no sense for an endpoint that is intended to be used by machines for testing. The endpoint now returns JSON only unless a redirect action is triggered.

# This is the commit message #1:

Fix the activation email support link in the dashboard sidebar

# This is the commit message #2:

Mask grades on progress page according to "Show Correctness" setting.

# This is the commit message #3:

More celery logging

# This is the commit message #4:

Eventing for grading policy change

# This is the commit message #5:

More specific grace_course logging

# This is the commit message #6:

Add course overrides of waffle flags.

# This is the commit message #7:

Mark test as flaky, and remove flaky from a fixed test.

EDUCATOR-511

# This is the commit message #8:

Fix for LEARNER-859: Updating styling on the course updates page to more clearly differentiate between multiple updates. Specifically:
- Updated styling on date in the top left corner
- Added horizontal line between updates
- Removed ability to toggle updates on and off
- Cleaned up code to always show all updates:
kdmccormick added a commit that referenced this pull request Jun 15, 2017
mumarkhan999 pushed a commit to mumarkhan999/edx-platform that referenced this pull request Mar 4, 2019
…to-caliper-app

move caliper config and processor from track app to caliper_track app
yasir1brahim referenced this pull request in yasir1brahim/edx-platform Apr 19, 2021
Updated the python-dateutil version for correct arrow installation

Approved-by: Rafay Khan
lpm0073 referenced this pull request in grid-synergy/edx-platform Apr 27, 2021
Sujeet1379 pushed a commit to chandrudev/edx-platform that referenced this pull request Nov 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants