Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MarlinDev as a separate project? #2578

Closed
thinkyhead opened this issue Aug 1, 2015 · 48 comments
Closed

MarlinDev as a separate project? #2578

thinkyhead opened this issue Aug 1, 2015 · 48 comments
Labels
Needs: More Data We need more data in order to proceed T: Development Makefiles, PlatformIO, Python scripts, etc.

Comments

@thinkyhead
Copy link
Member

@Wackerbarth, @fmalpartida, and I have been discussing how best to provide separation between development, code integration, release candidates, and releases. Among the ideas being floated is to create a separate project (MarlinFirmware/MarlinDev) where all development would be done, while keeping Integration and Release branches in the current repository. Richard can articulate better than me the added benefits of this approach. One of the main ones is separation of issues and PRs.

In any case, it would require @MarlinFirmware to move all the active @MarlinFirmware/collaborators, @MarlinFirmware/owners, and @MarlinFirmware/testers over to the new repo.

Let's discuss this concept.

@thinkyhead thinkyhead added Needs: More Data We need more data in order to proceed T: Development Makefiles, PlatformIO, Python scripts, etc. labels Aug 1, 2015
@avluis
Copy link
Contributor

avluis commented Aug 1, 2015

This is actually a very to thing to consider - just the separate issue tracking for the Release branch and Dev branch is enough reason to do this.

@boelle
Copy link
Contributor

boelle commented Aug 1, 2015

no need to move people... the different teams are all top level

@boelle
Copy link
Contributor

boelle commented Aug 1, 2015

repro created. access should be the same. let me know if not

@boelle
Copy link
Contributor

boelle commented Aug 1, 2015

could any of the branches here go? just so there is less messing and confusing here

@Wackerbarth
Copy link
Contributor

@boelle -- Would you please change the default branch for MarlinDev to "master"?

@boelle
Copy link
Contributor

boelle commented Aug 1, 2015

yep first thing in the morning

just gone to bed

Den søndag den 2. august 2015 skrev Richard Wackerbarth <
[email protected]>:

@boelle https://github.com/boelle -- Would you please change the
default branch for MarlinDev to "master"?


Reply to this email directly or view it on GitHub
#2578 (comment)
.

@thawkins
Copy link
Contributor

thawkins commented Aug 2, 2015

I don't have any issues with this as long as somebody is going to take the
responsibility on of moving distinct snapshots or release integration
branches from the dev repo to the release repo. (release engineering role)

are we also planning to provide a library of downloadable zip files for key
milestone tags in the releases repo, so users can avoid having to setup and
install git to get the code for a release.

I would also like to raise at this point that while we are looking at
methods to simplify things for endusers, it may be worth considering the
following.

  1. conduct a survey of users (method yet undetermined) to work out what the
    most common configurations are and support prebuild .hex files for these.
  2. move some of the customization to runtime commands stored in the eeprom
    with m500 etc.

Printrbot does this successfully, ther have a universal .hex file for all
their models. with the ability to set bed size and extruder offsets for
autoleveling. They precompile for two extruders even on a single extruder
machine. I would expect that we can come up 4-6 configs that given 2)
above could hit at least 75% of the user base. LCD displays are probably
the most problematic aspect. People that don't fall into those prebuilt
configs would have to rebuild as they do at the moment.

On Sun, Aug 2, 2015 at 5:39 AM Scott Lahteine [email protected]
wrote:

@Wackerbarth https://github.com/Wackerbarth, @fmalpartida
https://github.com/fmalpartida, and I have been discussing how best to
provide separation between development, release candidates, and releases.
Among the ideas being floated is to create a separate project (
MarlinFirmware/MarlinDev) where all development would be done, while
keeping Integration and Release branches in the current repository. Richard
can articulate better than me the added benefits of this approach.

In any case, it would require @MarlinFirmware
https://github.com/MarlinFirmware to move all the active
@MarlinFirmware/collaborators
https://github.com/orgs/MarlinFirmware/teams/collaborators,
@MarlinFirmware/owners
https://github.com/orgs/MarlinFirmware/teams/owners, and
@MarlinFirmware/testers
https://github.com/orgs/MarlinFirmware/teams/testers over to the new
repo.

Let's discuss this concept.


Reply to this email directly or view it on GitHub
#2578.

@boelle
Copy link
Contributor

boelle commented Aug 2, 2015

"@boelle -- Would you please change the default branch for MarlinDev to "master"?"

done

@boelle
Copy link
Contributor

boelle commented Aug 2, 2015

again what branches should we keep here? just so we have less clutter to confuse people and not doubles etc

@thawkins
Copy link
Contributor

thawkins commented Aug 2, 2015

Branches will be driven by need, if we have new features that will take
time to sort out then they will get a branch.

The only two we probaly need at the moment are master and a integration
branch, when a candidate tests out in the integration branch, it would be
merged into master on the release repo and then tagged.

On Sun, Aug 2, 2015, 14:49 Bo Herrmannsen [email protected] wrote:

again what branches should we keep here? just so we have less clutter to
confuse people and not doubles etc


Reply to this email directly or view it on GitHub
#2578 (comment)
.

@boelle
Copy link
Contributor

boelle commented Aug 2, 2015

ok let me ask more clear:

what do we need in each of the repro's ?

sure some of what is in Marlin no longer needs to stay as its now in Marlindev

@Wackerbarth
Copy link
Contributor

On Aug 2, 2015, at 2:38 AM, Bo Herrmannsen [email protected] wrote:

what do we need in each of the repro's ?

sure some of what is in Marlin no longer needs to stay as its now in Marlindev

At the present time, Marlin (is moving toward) has only 2 branches.

The “Release” branch serves as a “front page". But rather than being a “branch”, it acts more like a variable tag.
At all times, it will point to the latest tagged release. (Perhaps plus one commit involving the project’s README.md)

The actual branch which has a history of all of the releases in a series has a name of the form "..x" (eg "1.0.x”)

After we reach the “1.1.0" release, “Release” will point to that. There will also be a “1.1.x” branch. It will point to the 1.1.0 tag PLUS any approved patches. And the “1.0.x” branch will still be available.

At the present time, “Development” still remains in this repository ONLY TEMPORARILY while we clean up, migrate, or otherwise dispose of the existing Pull Requests.

Because of the way GitHub handles Pull Requests and merges, we may also create temporary “Inbox” branches against which patches may be submitted. But the important thing is that only the “Release Engineering Department” will make changes to the “Release”, “1.0.x”, “1.1.x”, etc. branches. And ANY OTHER branches are “temporary” and “transient”. They are subject to deletion at any time.

On MarlinDev, the same philosophy applies.
“master” is the current “front page” and “inbox” against which submissions will be made.
“dev” points to the current development and the “1.1.x”-type branches will be created as appropriate.

NOTE: Both of these repositories are actually just separate views of one underlying “master” repository. Their contents are both parts of a common git tree and it is trivial to move material from one to the other using simple git push commands from your local copy of the master copy. Each of these GitHub projects appear as a “remote” to your local system.

When, hopefully very soon now, we are ready to pre-release “1.1.x” for testing, we will expose it under the “RC” branch on the “Marlin” project. That way, feedback on testing, etc. can be kept separate from development taking place on “MarlinDev”.

@boelle
Copy link
Contributor

boelle commented Aug 2, 2015

so at the moment i should not delete any branches?

@Wackerbarth
Copy link
Contributor

Not on GitHub. I've already taken care of that aspect.

@boelle
Copy link
Contributor

boelle commented Aug 2, 2015

:-D

me just a noob here

btw. behind the scenes i have encouraged @Roxy-3DPrintBoard (aka Roaxana) to give a go at cleaning it up. She have done a lot of work but might need help on how to make a PR the right way

@Roxy-3D
Copy link
Member

Roxy-3D commented Aug 2, 2015

btw. behind the scenes i have encouraged @Roxy-3DPrintBoard (aka Roaxana) to give a go at cleaning it up. She have done a lot of work but might need help on how to make a PR the right way

Specifically... I was going to try to clean up the G28 & G29 code. There is a lot of stuff to clean up, but that is probably the most outrageous and it is an area that interests me. Perhaps we should start a separate thread to get people's comments (and help) on that topic. I guess I'll do that over in the MarlinDev area.

@boelle
Copy link
Contributor

boelle commented Aug 2, 2015

Good idea

One easy but maybe cumbersome way to do a PR is to fork, make changes... submit PR....

PR can be submitted from the web interface and also the PC based clients...

@boelle
Copy link
Contributor

boelle commented Aug 2, 2015

btw... i do only watch a very limited number of issues or my inbox get hammered full

so one way to get my attention is to do @ in front of my username... unsubscribing from this one also

@thinkyhead
Copy link
Member Author

The one thing I can think of that makes this a little trickier for contributors is that all the current forks of Marlin are based on this repo and not the new one. So the new repo adds an extra layer of complexity for users who want to submit customizations from their existing forks. And, we will have to tell everyone who submits a PR now that they must create a fork from the new repo and submit their PRs to the new repo.

Should we follow up on the several PRs that exist in this repo, merge them here and then bring them over to the new repo. Or, should we post a message on all PRs that they should redo them with the new repo?

@Wackerbarth
Copy link
Contributor

Yes, it is a slight complexity for the small number of individuals who are working with "before pre-release"

However, I anticipate that we will do "Testing" here in this copy and merge those resulting changes here and copy them "internally". But those details will need to be determined as we gain some experience with the setup.

As to open PR's. Let's close (with a message to resubmit on MarlinDev) any that are old.
And let's merge (here) any that you find ready, or with only minimal adjustment. Then we can re-evaluate the ones that remain in that gray area between the two extremes.

@boelle
Copy link
Contributor

boelle commented Aug 3, 2015

closing this one as it has been done

@boelle boelle closed this as completed Aug 3, 2015
@AnHardt
Copy link
Contributor

AnHardt commented Aug 3, 2015

Reopened because this explains a bit what is going on now. (At least for a week or two)

@AnHardt AnHardt reopened this Aug 3, 2015
@a4jp-com
Copy link

a4jp-com commented Sep 7, 2015

Even if this topic is closed, people can still write here can't they?

@AnHardt AnHardt closed this as completed Oct 14, 2015
@Roxy-3D Roxy-3D reopened this Nov 25, 2015
@Roxy-3D
Copy link
Member

Roxy-3D commented Nov 25, 2015

I would appreciate a short discussion on what we did here. If you have strong feelings or good insight, please do not hold back.

Specifically, I'm a little bit concerned we made extra work for our contributors by doing this. And in fact I think it is possible this split caused one of our repositories to get messed up because of the extra complexity it added to the environment. (Big Thanks to Wackerbarth for cleaning up the mess!!!!)

Are we getting the benefits we anticipated? Is the separation between Marlin and MarlinDev really giving us benefit? Especially once we promote a Release Candidate to 'Stable' status?

@Wackerbarth
Copy link
Contributor

@Roxy-3DPrintBoard -- We did this for two reasons.

  1. We wanted to try to separate "support" issues on Released firmware from "development" issues.
  2. There are significant differences in file structures, build instructions, etc. It was felt that some separation would help keep individuals from attempting to apply the wrong set of instructions.

And remember that we were not supposed to have this RC period drag out "forever".
The intention was that Marlin would become an almost static "site". The "extra work" related to having any stability in releases and still allowing active development will always exist. And, if you get away from using the github pull requests for the back porting, it really isn't very bad.

IMHO, misusing the github workflow, whether in one repository, or many, is the source of much of the hassle.

@Roxy-3D
Copy link
Member

Roxy-3D commented Nov 25, 2015

OK. Two questions for @Wackerbarth :

  1. When the RC period closes and we promote something to a 'Stable' status, does the reason for the split go away in your mind?

  2. Is the split really helping to separate support issues from development issues? I'm OK with the way it is right now, but there is a lot of chatter on each side that really belongs on the other side. What we have done is made it so the contributors here need to check and be active in two areas instead of one.

@Wackerbarth
Copy link
Contributor

  1. No, not at all.. In fact, it becomes even more important. The instructions for building are different, features and settings will be different, etc. We will have most less-knowledgable users trying to rebuild their firmware. So the separation is really more important at that point. Additionally, the "I can't get .... to work" issues remain separated from the actual submissions which add to the code base.

  2. I don't think that we are realizing many of the benefits of the split yet. There is still too much "development" being attempted in the Marlin repository.

@AnHardt
Copy link
Contributor

AnHardt commented Nov 27, 2015

Two Repositories.

In the order of convenience i use the GitHub-Web-interface, the Github-desktop-program, git gui, gitk and the command-line git.
My normally save way to handle a GitHub repository is to fork, for example MarlinFirmware/Marlin to AnHardt/Marlin, clone AnHard/Marlin to my computer. To distinguish between the 3 levels i can use 'origin' - for AnHard, 'upstream' for MarlinFirmware and nothing for local.
Now for MarlinFirmware/MarlinDev -> AnHard/MarlinDev -> ???.
Use a second local folder? Then i can't transport commits from one to the other repository.
Or make AnHard/MarlinDev a second source in the first folder? Then i can't use 'origin' and 'upstream' because it is not clear what is mend - Marlin or MarlinDev. Suddenly i have to deal with 4 similar looking web-addresses to type or paste on the command line and a confused GitHub-desktop-program. It's proved I do make errors then.
So the current state for me is to have 3 local repository. One for Marlin, one for MarlinDev and one to let them interact.
In total that's dealing with 7 repository instead of 3. A lot of work to keep them in sync. Constantly you have to answer the question - Where am I.

I know, searching for something is a forgotten art. Just to ask around and waiting for someone else to do the work is a common attitude. In some cases the one who answers the question is knowing the fact by hart - tat's luck. In the other case you have someone actively searing in the available sources. A experienced supporter has usually an idea how to narrow down the results, (author issue, PR, timespan, keywords, ...) With two repository the supporter additionally has to remember in witch one - or has to do the search twice.

References. Instead of just typing #2797 you now paste "#2797" if in the other repository.

Mailfilter -> doubled

...

It piles up

Any tips?

@thinkyhead
Copy link
Member Author

As this is now a foregone conclusion, I think we can close this initial topic. It may be worth revisiting in the future. For my part, I've given in (!) and I'm finding it more convenient to have the "Marlin" repo only dealing with releases, while the "MarlinDev" repo only deals with development towards 1.2.x. The main inconvenience is of course keeping the two source-trees in sync and having to make 2 PRs for every bug-fix, and keeping issues in the right places. Fortunately, so far the sources have not diverged significantly, so we can still compare the sources and catch any patches we missed earlier. Some duplication of effort will probably always be unavoidable, as in any project there are going to be bugs discovered in the next version which you'll want to apply to a point-release on the previous version.

@Wackerbarth
Copy link
Contributor

It wouldn't be such an issue if the two of you would stop trying to add improvements in the 1.1 branch.

We certainly know that there will always be room for improvement. But we really need to simply "draw the line" and fix ONLY the most egregious problems in the 1.1 branch.

@thinkyhead
Copy link
Member Author

It wouldn't be such an issue if the two of you would…

Geez, you try to be conciliatory and you get fingers pointed in your face.

The release will take as long as it takes. I think we all understand that. While there are still some bed-leveling and other "egregious" issues extant, it's no skin off our nose in the meantime to include some of the little improvements being made along the way. I may bitch too much about it, but the main bottleneck is not the extra effort of making two PRs for every patch (which we would have to do for 1.1.1, 1.1.2, etc. anyway). The main bottleneck is that everyone is doing this in their spare time, so we do a lot of waiting for testers and patches.

Anyway, I did almost zero on this for a couple months. Now I'm spending a couple hours a day. This will get things moving a lot faster, even if there are a few extra PRs involved. So count yer blessings!

@a4jp-com
Copy link

I think the message from @Wackerbarth was a bit of a joke. You're doing a great job and any help is greatly appreciated. Do you need any help testing anything?

@Roxy-3D
Copy link
Member

Roxy-3D commented Feb 11, 2016

@thinkyhead Everything is good... Please keep help fixing bugs!

@a4jp-com
Copy link

Just out of interest, how is the bug count going now?

@thawkins
Copy link
Contributor

I would like to ask if somebody could do a quick write up on a wiki page of
the process from fix to PR etc, i tried to do it once and got confused and
brow beaten about it, so i decided not to bother again, i suspect you folks
would get a lot more help if the onboarding process for devs was easier to
discover.
On Feb 11, 2016 12:37 PM, "Scott Lahteine" [email protected] wrote:

It wouldn't be such an issue if the two of you would…

Geez, you try to be conciliatory and you get fingers pointed in your face.

The release will take as long as it takes. I think we all understand that.
While there are still some bed-leveling and other "egregious" issues
extant, it's no skin off our nose in the meantime to include some of the
little improvements being made along the way. I may bitch too much about
it, but the main bottleneck is not the extra effort of making two PRs for
every patch (which we would have to do for 1.1.1, 1.1.2, etc. anyway). The
main bottleneck is that everyone is doing this in their spare time, so we
do a lot of waiting for testers and patches.

Anyway, I did almost zero on this for a couple months. Now I'm spending a
couple hours a day. This will get things moving a lot faster, even if there
are a few extra PRs involved.


Reply to this email directly or view it on GitHub
#2578 (comment)
.

@a4jp-com
Copy link

What is the address of the Wiki page you made?

@AnHardt
Copy link
Contributor

AnHardt commented Feb 11, 2016

@Wackerbarth
It wouldn't be such an issue if the one of you would not had not stopped MarlinDev to compile in a 'normal' way.

@a4jp-com
Copy link

The version I have compiles properly. Which version is the version that doesn't compile @AnHardt? Would you like a link to my version?

@Roxy-3D
Copy link
Member

Roxy-3D commented Feb 13, 2016

It wouldn't be such an issue if the one of you would had not stopped
MarlinDev to compile in a 'normal' way.

So... Just for the record.... I'm not doing this.... I am developing
with the new file organization in MarlinDev, but I'm not using this
feature.

If you want.... You can set the #defines so it uses your old
Configuration.h file. You don't have to do the inheritance thing.

On Fri, Feb 12, 2016 at 7:15 PM, a4jp-com [email protected] wrote:

The version I have compiles properly. Which version is the version that
doesn't compile @AnHardt https://github.com/AnHardt? Would you like a
link to my version?


Reply to this email directly or view it on GitHub
#2578 (comment)
.

@Wackerbarth
Copy link
Contributor

@thinkyhead - "The release will take as long as it takes. I think we all understand that. While there are still some bed-leveling and other "egregious" issues extant, it's no skin off our nose in the meantime to include some of the little improvements being made along the way."

I cannot agree. It is "skin off our nose" when you introduce these "little improvements" because that creates an ever moving target. Unfortunately, even though the "improvements" are each well intentioned, there are many subtle interactions which are never fully vetted before they get introduced into the "product". That, IMHO, is why we are having the problems with bed leveling at this time. If, during the correction to these problems, you continue to introduce additional "improvements", the chances of having some additional problem introduced keep increasing. And, without a stable target, all of the testing that is done remains incomplete. As such, you will NEVER reach a stable, well tested, version that is worthy of being called a Release.

Therefore, if you want to introduce improvements, you really should limit them to the code base which will be a part of a future release cycle.

If, in your opinion, the current release candidate is so flawed that you feel that additional improvements can be added before the major problems are addressed, then we should consider abandoning the current release candidate and jump directly to the next cycle. But doing that implies that we include changes which come from contributors other than yourself. That code base already exists in MarlinDev.
Were it not for my reluctance to introduce some significant changes to a release cycle in progress, we would have already made the initial jump to the transitional version.

You should also note that we have actually made two major changes from the code base structure of RC3. I made a deliberate effort to provide a transitional path which introduces them in stages so that it would reduce the incremental complexity for a user to migrate over.

@thinkyhead
Copy link
Member Author

At the moment the number of actual BUGS being reported for RC is quite low. I think we're getting very close to release. RC4 might just be the last RC.

@Roxy-3D
Copy link
Member

Roxy-3D commented Feb 19, 2016

That would be nice if it turns out to be true! The Unused End-Stop / Undeployed Z-Probe problems account for about half of the problems I've seen posted. And AnHardt has some Pull Requests for that set of problems that have been very thoroughly vetted. We mostly just need to get them joined in with the rest of the bug fixes and make sure everything works as expected.

And the other issue that is very serious is the flaws in the Serial Communications where characters can get dropped. That is very serious from a user's perspective because it can stop a print at a random time with no explanation. From what I now understand about the problem, I think it is critical we get those Pull Requests into the RCBugFix also.

If we can all agree those things should be done... I'll load up RCBugFix as soon as its ready and start beating on it.

@thinkyhead
Copy link
Member Author

flaws in the Serial Communications where characters can get dropped

Can that actually be prevented or are we stuck just mitigating it through the use of a checksum, OK responses, etc.?

@AnHardt
Copy link
Contributor

AnHardt commented Feb 19, 2016

Na. All the lost characters are detected if the line has a checksum. And is requested to resend. Normally the worst thing is a momentary stop of the head.

@AnHardt
Copy link
Contributor

AnHardt commented Feb 19, 2016

Yes. The overflows are away when sending the correct number of ok's. But there is another error where 'strange' strings, never sent by the host, appear in the input stream. I have no better explanation for them than a race condition between the normal reading program and the two writing interrupts. And they vanish with my patches.

@thinkyhead
Copy link
Member Author

they vanish with my patches.

That's what we want! In the long run, I'm sure a flowchart will be a great help. Then we can authoritatively say "this happens at this time," and "that happens at that time." On the other hand, looking at other firmware, like Repetier, can also be instructive!

@Roxy-3D
Copy link
Member

Roxy-3D commented Feb 19, 2016

Na. All the lost characters are detected if the line has a checksum. And is requested to resend. Normally the worst thing is a momentary stop of the head.

I'm just nit-picking here. The 8 bit checksum is very weak. If we lose two characters it is possible for them to cancel out each other's effect on the checksum. Specifically, for example, suppose we lose two 0's in a number. The checksum would calculate to be correct. There isn't much we can do about it.

But there is another error where 'strange' strings, never sent by the host, appear in the input stream. I have no better explanation for them than a race condition between the normal reading program and the two writing interrupts. And they vanish with my patches.

I believe this 'race condition' was real. It hardly ever happened, but without the patches it could happen at any time. And the strange data (left over from previous commands) is exactly what would happen.

Then we can authoritatively say "this happens at this time," and "that happens at that time."

It is difficult to make happen, but getting an interrupt in the middle of updating those Rx and Tx buffer indexes definitely could cause problems because other code (the interrupt handler) would be using an incorrect value for the Rx and Tx buffer indexes.

@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked and limited conversation to collaborators Apr 12, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Needs: More Data We need more data in order to proceed T: Development Makefiles, PlatformIO, Python scripts, etc.
Projects
None yet
Development

No branches or pull requests

8 participants