Community improvements and pipeline for the official Blender Developer Tracker
Issue
To create feature requests that can ultimately become tasks a developer can work on, commit to, and do... is very long an buerocratic process.
Here is the current method to be heard in the Blender community and talk with developers directly:
- Post in Rclickselect, a third party voting system.
- If you get enough votes, maybe you can go to Devtalk, a forum for developers.
- If you get enough attention in the forum, then you can then go to the official Developer Tracker.
- Once you get to the tracker, you can only post bugs, or actually do the work and post patches to the very proposal you made.
- If you can't do it yourself, you can go to the Blender Chat and talk about it - but it's not a task for the master, a branch, or patch.
- If all else fails, you can go to youtube comments, third party forums, and social networks to try talk to individual devs ... and be drowned out by disorganized noise. Discord, Blenderartists, Facebook, Twitter, Youtube, etc.
- And then, once you are done trying to reach out, you can then make your own repo on Gitlab or Github and commit to your own patch, then wait for approval or traige of your committed patch in the official Developer Tracker.
Being a Blender developer is no joke, it's becoming a politician, it's becoming the lead dev of your very own proposal, it's daunting.
Where do people of the community become part of development here? Only bugs and/or actual code? What about feature requests? What about troubleshooting? What about direct feedback to devs that can then turn into tasks that will ultimately merge to Master?
There is a logistical canyon between Blender developers and Users.
Why is this deficient?
- Feature requests and feedback suddenly became politically and financially motivated.
- Talking with developers requires 3+ distinct un-official dialogue channels with developers, and even then it's multi-layered with an unknown hierarchy.
- There are no "tasks" in the tracker, only "bugs", "to do", and "design" (by tag).
- There are no "feature requests" in the tracker.
- When you propose any feature request in the tracker, it risks being disregarded, closed, or marked as invalid due to not adhering to the tracker standards of "bugs", "to do" and "design", being redirected to the convoluted suggestion process above.
- Discussion with developers is unofficial and all over the place.
- To propose something "to do" or "design" requires political prowness to direct developers, and often due to lack of puiblic hierarchy in the Blender Foundation (90% of development), these tasks can and most often will be marked as invalid - and thus the birth of all these third party systems for the market-to-producer relationship.
- There is no organized system of making feature requests that can transition into into actual developer tasks to be branched, committed to, merged and triaged by other developers in the same developer space.
- There is a disjointed non-transparent connection between feature requests, the community ideas and what developers do.
- Transparency is laborous and requires public relations (youtube updates, forum updates, developer social updates, community management, etc) thus increasing disgruntled customer service.
- You can ask for something, propose something, and have it lost before word gets to a developer working in the tracker due to all the above listed.
- The code committing system also needs to be worked on. Right now you can propose patches, which are third-party files, wait for review, then maybe it get's pulled in (if it's approved by lead devs of the Blender Institute only).
- Making your own branch has little or no guidelines.
- There are no branch pull requests revision system so Blender branches are all over the place (graphicall, Rclickselect, private studios, private and public forks, etc)
- I would even consider patches redundant, as a task can hold commits to code from a branch (and tasks other tasks and their commits), and a pull request can house all commits for a project.
The system is convoluted, misnamed (maniphest is.. the tracker?), and difficult to master - documentation is sparse and dense, no artist friendly guides or GUI systems to work with, hierarchy is hard to understand, feedback is disjointed, and the pipeline mysterious to the point that understanding development is only done through updates based on forums, chats, socials and videos.
I admit, I personally haven't ever been able to commit, branch or work with Blender directly. I have developed personally for a fork of Blender on Github with it's own distros and developer ecosystem, even learnt to compile there BEFORE Blender, as a noob, and it has been an easy, transparent and engaging experience: being an artists slowly becoming more of a developer for other artists there. I have adventured into the Blender tracker with it's 3000+ tasks organized in a system that's not easy to understand, hard to filter, with heavy console based documentation... I could be biased.. but for the vision of "onboarding" - the very fabric of opensource development needs to be addressed - and at it's core.. the developer tracker and workflow
Solution
In Github you can have tags, tasks, commits, branches, and pull requests into a clearly defined Master/branch relationship. And these tags can be already technically be layered in the Developer Tracker also.
Right now the developer tracker tasks don't have enough tags based on workflow/pipeline/community.
I suggest modifying the current tracker tag system to allow for more community involvement and organization of community data, ideas, needs and involvement directly in the Blender Tracker with a workflow and feedback type of way.
I also suggest the tracker has a role based system, as you can do in Github also.
And ultimately, I suggest making a workflow to commit and work on Blender sourcecode follow a standard like that of Github or Gitlab.
In General, the development tracker needs to be standardized as much as the Blender software is trying to become industry standard.
Mockup:
1. Roles
-
Community Users:
- Create tracker issues
-
Permitted labels in only one of the following:
- Feature Request
- Help Wanted
- Question
- Bug
- Cannot relabel issues
- Cannot close issues
- Can vote on any issue
- Can only talk on their own issues or other community issues (with the community labels).
-
Blender Foundation Developer
-
Full control to close issues and relabel issues
- The former
- Task
- Fixed
- Bug
- Duplicate
- Needs Work
- Post-poned
- Won't Fix
- Works as Designed
- Invalid
- Full control to assign developers to issues
- Full control to create tasks
- Full control to close tasks
- Can vote on any issue
- Can create and approve pull requests
- Can branch and commit to issues (per project branches)
- Can branch and commit to issues (per developer named branches)
- Can talk on all issues
-
Full control to close issues and relabel issues
-
Third-Party developer
- Partial control to close issues and relabel issues
- Partial control to assign developers to issues
- Partial control to close tasks
- Full control to create tasks
- Can vote on any issue
- Can create pull requests only
- Can branch and commit to issues (per developer named branches)
- Can branch and commit to issues (per project branches)
- Can talk on all issues
-
** Sponsored Third-Party developer**
- This developer would have the reputation and ability to work on Blender outside the Blender Foundation but have the same power as a developer funded by the Blender Foundation.
- He can approve pull requests and merge to master, basically
- Like a Blender Foundation Developer, he also has full control.
2. Pipeline tags
-
A community user task now can be directly reviewed by the Blender Institute and/or a Third-Party developer
-
Once it's reviewed, it can be labeled accordingly by a role that can.
Community Labels that would be changed: - Bug = Task + Bug if it's a confirmed bug that needs to be fixed - Feature Request = Task if it it's a confirmed feature request that now can be a task to be implemented - Help Wanted and/or Question = Fixed if the question or help wanted is solved
-
Once an issue is turned into a task, a developer or third party developer can assign different states that can be used to get the task done:
- 0. Task = is an issue from the community turned into a task that now is assigned to a developer to be done, or put into a kanban queue state. Developer can also do the same, propose an issue, confirm it's a task, then get it assigned and done.
- 1. Fixed = shows that an issue or task is fixed or closed. Here it can show what needs to be reviewed pulled into master, if things are under revision to be pulled, it might be fixed, but not closed and merged to master yet.
- 2. Bug = shows that a task is bug related, not feature/design related
- 3. duplicate = show that this has been linked and repeated to another task, ultimately closing this
- 4. needs work = shows that this task requires more effort to make it able to be pulled into master
- 5. Post-Poned = a very important one for the community. There are 3000+ tasks, but many have been there for years and we don't know what's up. Is it being worked on? Is it dead? has it been reviewed? This would be used to show that a task has been noticed, but it will have to wait for future release. The devs and minimize focus and put a task on the backburner. This is for the backlog.
- 6. Won't Fix = also important for community feedback. Some bugs or features to fix something are just out of scope or not in the Blender vision. Adding this has a quick explanation to a dev that this task should not be done, though it might be correct. Thus closing.
- 7. Works as Designed = similar to the previous, so features are part of the Blender vision making the task invalid. It's good to give that feedback.
- 8. Invalid = some tasks from devs or community are.. just not valid. Not enough info, too abstract, too ambitious.
-
Once the task has either been resolved or moved to a state worthy of closing, the developer can then close, pull, and then wait revision from other developers to merge to master.
3. Other Task Category Tags
-
Some of the tags in the Blender Developer tracker is used to assign what area of the Blender Institute is getting work, this can also be done with tags like the following, they would added to the above tags. eg
Task + Bug + Website
:
- Website
- Forum
- Promotion
- Cloud Service
- Etc
4. Pipeline
Like Github, the pipeline will open up documentation and portals to do the following (some of this functionality is already exposed in the Blender Tracker, but it's missing clear tutorials and documentation for the common man):
-
As a Third Party Developer (once registered), you can do the following:
- Create a personal Branch to commit assigned tasks
- Have the personal branch available for pull requests on a per user basis
- Be able to be assigned to project branches and commit to project branches
- Self assign to tasks
-
As a Blender Institute Developer
- Create a personal Branch to commit assigned tasks
- Have the personal branch available for Pull requests on a per user basis
- Be able to be assigned to project branches and commit to project branches
- Approve Pull requests and review Project Pull requests and user branch based pull requests
- Self assign to tasks
-
As a community member.
- I can see everything in a user friendly way and feel involved in what I'm vested in directly with the developers.
- My voice can be directly heard by those who have the power to improve what I love and use every day
- If I back the project financially, I can get my moneys worth for the subscription I pay to the institute, from customer support instead of just third-party community support.
-
User friendly documentation and guidelines based on typical Gitlab or Github pipelines
- Everyone is used to this workflow: task, branch, commit, pull, review, merge, start again from a sync from master.
- Standards will be easy to pick up in a couple of minutes
- Code will be organized and clear based on developer branches or project branches
- The workflow of how code is created, committed and reviewed will be clear and no-longer mistified by "patches".
- Commits and pull requests (aka, patches) will be open to discussion, vote and be a part of the tracker instead of.. an abstract page to wait for review. The pipeline in the current tracker is.... unstandard and difficitul to master.
- Easier lists of branches and project, progress meters, developer branches, and filters so that it's easy to see what is happening and where, and how to help.
Why?
Blender wants to onboard artists and developers, what better way than making the UX of developing and community feedback the one and the same and in an organized, useful way - INSIDE THE BLENDER TRACKER. Github allows for this, the Blender tracker should too.
- The tracker could become what Rclick select is, with additional votes, but unified with developer votes and opinions.
- The tracker would also become a system of status updates based on the tag system and the filters, with a user friendly system for feedback and organization of ideas.
- The tracker now is more Agile and Scrum facilitated with status of a backlog, idea of what is considered, and what is not, what is done and being worked on, etc. This will help organize the 3000+ current tasks that clearly has shown huge inefficiency in development.
- The workflow would be more involved with third-party developers, thus more opensource and not monopolized by the Blender Institute, risking user benefits due to lack of being.. opensource.
- The code pipeline is no-longer alien to typical devs and artists (who works in patches!?).
- Communication and feedback (and thus software quality and development speed) will be directly optimized.
- Community roles and feedback directly in the tracker and on bugs in the tracker through their permitted tags can optimize work, but not pollute the tracker with noise as you would have to be a developer to have access to talking on tasks directly.
There is no need to shut down these existing communication portals and feedback areas - but the tracker needs to be opened up to the community and made to become part of the development pipeline, directly. It has to be easy, for everyone.
17 Comments
(Disclaimer: I started RCS, so I probably have some level of bias)
There are a few problematic assumptions in this proposal. First and foremost, it assumes that the Blender core development team isn't aware of the features and improvements that should be in Blender... and that they don't already have lists and priorities on what needs to be developed.
The second (and potentially more problematic) assumption is that of time. Here on RCS, we get roughly 4-5 new feature proposals every day. I read every one of them. I respond to most of them. I am not a Blender developer. Now, on developer.blender.org , we already know the time-consuming process of triaging bug reports and reviewing submitted patches. There's already a backlog there. Would you rather have developers fixing bugs and reviewing patches for features/improvements that have already been written (in addition to writing their own code), or would you rather them spend their time doing what I do here? Because adding the load of this site to developer.blender.org would necessarily decrease the time spent on those other three tasks.
That said , one of the long-standing features for RCS that I've wanted to have implemented is the ability to link a proposal here to a design task on developer.blender.org when it gets to that point (and a cleaner way to do internal cross-links for related and duplicate posts... but that's another topic). That feature would provide a more natural bridge between this site and that. It may also be nice to extend moderator rights to developers here. But including feature requests in the bug tracker is pretty unwise, in my opinion... especially considering the percentage of posts here that are truly actionable proposals.
* @Jason van Gumster * Ultimately, the more users in the tracker, with restricted rights to not make noise, can ease on the public relations and organizes transparency and prioritizes work and gets more people developing, thus easing the workload on core devs ultimately. This voting thread for proposals has been a godsend, and I applaud it, and it has indirectly made every user feel like they are "developing" their software, but they are directly not. I am happy to hear that it will somehow link to the tracker, as it should, which is a step towards the proposal above.
But again, keeping it all fusioned with public relations with the userbase, will have the same devs have the roadmap exposed to the users, feature requests and user needs right in the task lists, and better self organizing of what needs to be done - it will be more open, as open source should be. Knowing a devs priorty and feature's they'll work on is... disjointed and hidden in the tracker, who knows what they will do!? It could be a post-it handwritten in the blender foundation whiteboard, or a to-do task disconnected from any direct feedback - making echo chambers of users feeling like they are not heard till they are pleasantly surprised by surprise features, or heavily disappointed. I know devs are swamped, but that's because the community is not involved in the dev work (+90% of the commits pushed to master is the Blender foundation, seriously). Having more involvement of moderators and community devs in the actual tracker managing feature requests instead of a disjointed communities shouting in various areas will make the tracker more efficient. How can 3000+ tasks be labeled only 3 states of work and be called efficient?
Even if you don't connect feature requests and user feedback to the tracker, the tracker itself needs better organization and labeling and filtering of tasks. The internal kanban for Blender versioning tasks is literally.. nearly impossible to access from the global or sub navigation, for instance.
Ultimately, a lead dev doesn't have to work on kanbans or public relations (any studio has extra people doing that, like you), and the system does need managers as stated often publicly - why not make the actual userbase and community developers do exactly that? As a community developer, with that role, you can organize and turn any feedback from the community into tasks to be done, or disregard, test, or pitch patches and direct solutions to the community feedback directly in a tracker system.. it will track everything. A tracker is designed for exactly what it's supposed to do, track everything, not only bugs and code, but work to be done and what is needed to be done, and why it needs to be done - which is all over the place right now.
Another eg. if any of these RCS proposals come to a conclusion, it would be closed or labeled accordingly, become a number of connected tasks or a project/branch with sub tasks, and it can be worked on transparently by and with the community - and/or either postponed for the upcoming release for another later release, or assigned to people in the community or in the foundation that are now.. funded by the very public who want their needs to be done and then distributed to others.
The idea of an open tracker and open management by the community is to do what you hope to do, ease the developers to do their thing (because tasks are better managed by the public with more community developers), instead of making it hard for the userbase to be heard and seen which is detrimental to any dev who works for the userbase ultimately.
The very fact that RCS needed to exist in the first place shows the shortcomings of how developers use the tracker, but I'm happy it exists. If this system fusions even a little with the tracker, that is a step in the right direction.
Not being a developer; I tend to agree with this sentiment. It would seem that a transparent way to understand where developer focus is or is going to be in the near future, would enable better feedback from users at the times it is most useful.
Though I also understand that at times; too many people try to re-invent things that could otherwise have simple fixes. I would prefer that simple fixes were given priority with other more involved changes put into a separate channel.
I also agree that finding the correct way to report a bug or feature request is likley lost on many new users. It took me a couple years to really get a solid understanding of what wouldn't get imminently ignored or closed.
As someone who only recently started entering he community, I have pretty much the same sentiments. Just figuring out where to post a bug report took me a good 40 minutes of looking for scattered information that in many cases doesn't complement and contradicts each other. You have two places that perform more or less the same functions (devtalk and RCS) and two where you can post bug reports (developer.blender and devtalk). That by itself is redundancy that complicates the life of people who just want to suggest or report a bug without having to learn how Blender structures its development process. That's equivalent of requiring a driver to know how a clutch works before teaching the person how to change gears.
An example of that is how some people are told to post some requests on reddit instead of here, or are pointed at devtalk instead. Where is their request going to? Will any developer even notice it exists when there's no clear centralized place for such? I don't know about you, but aren't the whole community sites supposed to help developers have a more direct line with the target audience they are making features for instead of further distancing them from said target?
I wouldn't be surprised if real bugs that could've been fixed if noticed at an earlier stage just went by because the user who noticed them was either shut down at an earlier stage of reporting it or couldn't find the correct place to do and just gave up on the idea.
I personally feel that the Blender developers put much of the burden that should be made simpler or more practical on the end user, who in the end only wants the assurance that their concerns were heard and noticed and response, but ends up with a project that lacks the proper instructions instead.
* @bruno.souza.lino * Figuring out where to post a bug took you 40 minutes? It's right there in Blender's Help menu.
* @Jason van Gumster * some issues are not bugs. Features requests, technical help and ux issues have no official home in the tracker.
* @Andres Stephens - Trinumedia * And rightly so, IMHO. The tracker is for tracking development. Contrary to popular belief, developer.blender.org is not GitHub, nor should it be. GitHub uses their issue tracker for all of these things because there's literally nothing else there to use for those them... most of which are better addressed with other communication structures. People used what they had available on that platform.
I'm all for finding ways to integrate things more tightly with better cross-referencing, but I'm also a big proponent of using the right tool for the job.
* @Jason van Gumster * UX development, feature requests, technical help and all is in it's essence: development. Bugs are not the core basis of building a software. The portals for the former kind of development is still all over the place and needs to be centralized to be transparent and efficient. A tracker is a tracker. Not an exclusive bug tracker. It's a tracker. A software development tracker. Yes, not github, but a tracker. It should live up to it's name and track everything software development related, including user needs and requests, UX and troubleshooting. Or else it's a logistical nightmare (as it is right now) trying to track everything that is indeed software development related. This software exists for the user, there needs a method to track the user for development - and it's way too fragmented right now. If stackexchange, blender artists, blender.comunity and all were linked to the tracker tasks, with the ability to create tasks if things are confirmed as design tasks or bugs - then that would be great. But it would seriously be easier to just expand the use of the tracker instead of doubling logistical work all over the place and needing a team of managers just to run the de-centralized broken-telephone system of what we call "community development".
* @Andres Stephens - Trinumedia * I absolutely disagree. The tracker is already used for more than just bug reports. It also tracks development, design tasks, documentation, and code review.
The things that go on StackExchange, BlenderArtists, Blender.Community absolutely do not belong on developer.blender.org .
Let's look at BlenderArtists.org for a simple comparison. That site sees 150,000 views a day... roughly 550 new threads (not replies... just new threads) each week. Do you really want to flood developer.blender.org with that traffic and have the Blender developers triage each of them?
* @Jason van Gumster * If something is confirmed as a bug, a valid feature request, UX task, or other, (by the said community) then yes - it should directly link in. Not all 550 topics will go to the tracker per day, as they are not relevant - but relevant development topics should. Not everything created in a forum or chat system goes to the tracker. And if an issue already exists, it will be easily linked in to the userbase and get it's vote, community triage and more: a potential fix without a core dev. The idea is that these communities manage and run and work directly in the tracker instead of walking through hoops to help the core devs do their job so that nothing development related is lost - as many things do get lost right now. And this needs to either be automated, or allow to have these communities to co-exist with the tracker.
Example.. of the thousands of posts in the Paper-cuts thread on the Blender Dev forum, a simple system of allowing the comments and requests to become a task, or to have paper-cuts allowed to be triaged in the tracker directly - will allow the hundreds of ideas to actually exist and be developed instead of being lost in a forum in time. Allowing those who agree or disagree to triage the very requests in the tracker with or without a core dev, will make it efficient - and.. it will be easier to get a dev onboard to actually merge the fixes and proposals to master because they won't have to first go anywhere else to know what to do and for what audience.
Another example , on the stack exchange, a UI issue on default startup was posted, people talked of it, voted, a workaround done.. but it never was fixed or shown in a tracker. Is this responsibility of the user to post this twice or more than twice in multiple areas? Why not just in one and have it triaged directly? Then even fixed? Or better yet, have it provide a fix from the community directly to the source code directly in the tracker?
The user is the end goal of this software. Not the developer, we develop for the user.
150,000 views of potential developers triaging tasks and bugs with the core devs is a godsend. Not a curse. More people working on their needs is less work for a core dev. This is the spirit of opensource. Make it easy. Make it central.
You personally will have a bias, and that is respectable, but time will tell with more investment of studios, professionals, hobbiest and with the mess that things currently are will never make this software as idealistic as it aspires to be with an eventual disgrunteled userbase, and also.. disgruntled potential developers/investors - only increasing workload on core devs instead of sharing it with the very community the software exists for.
This proposal is essentially about adding tags to increase community involvement with the devs to speed development up. It's not revolutionary, and yes... this debate can be conceptual.. but seriously, getting an avalanche of disgruntled users asking for things and the fear behind that speaks directly to the quality of the software - if that is your fear, the quality of development and the connection with user requests/reports needs to increase one way or another.
Summery: This proposal is about adding tags to the current tracker, documenting it, and allow some functionality of other communities into the tracker - and to measure the mess with earned roles/restrictions and community lead curation to triage. All this is to ease development, optimize the source-code and apply solutions for the end user who is ultimately.. funding the Blender Foundation and allowing the software (and developers) to have a reason to exist.
* @Andres Stephens - Trinumedia * This is a fine theory, but in application I don't think your proposal is going to drive an increase in triage or community development as much as it's going to generate a lot of non-development-oriented distraction.
Look, for purely selfish reasons, I think it would be lovely to roll RCS into developer.blender.org . It would certainly free me up to do other things. But there are solid reasons why feature proposals have never been part of Blender's tracker. Those reasons have been hashed out before (and elsewhere), but if you'd like me to repeat them, let me know.
* @Jason van Gumster * I've had a discussion with other developer friends of mine for closed source software, even in areas that require your arguments as it's mostly cryptography (security software). I get the reason why you'd want to keep the tracker and triaging very closed and under a tight control under those who are the core devs. I get it.
Blender is too far in it's system to implement changes directly in a workflow that has been established for years at the source.
Devs are too busy to adapt or allow any new noise in the tracker as work is backlogged so much that it would be overwhelming to allow more into the tracker.
Triaging should be done exclusively by trusted developers to not allow any broken features, code, bug or malware into the source code or even into pull requests till developers have earned their trust and work with the current pipeline that already exists.
These reasons are all part of the explanation why a tracker has to be somewhat closed - as more often than not.. you only open a tracker issue for development of a feature request or UX issue per client request if the developer is financed/re-compensated (eg, client pays for it) or if it's clear the community areas require an issue bad enough (political enough) to convince a hard worked developer to ignore 3000+ other backlogged tasks to take on a new one people would democratically but maybe irrationally want (out of development ignorance).
This doesn't give me any good feeling to put in effort to develop Blender personally, though I do compile and fix bugs for myself and others in another Blender fork on Github, but I get it - and I guess we will see how things develop. If the fork stops needing to exist due to these control gates and more focus on what the fork tries to tackle, I will invest time into Blender directly - but till then I still can't be bothered working myself up the ladder to triage, commit, and work with the source for the amount of time I'd need to invest to even just learn how to commit and triage to merge/pull my own bug/feature report that I've already technically fixed. Learning the code fixes for Blender it time consuming enough.
* @Andres Stephens - Trinumedia * The thing is that it's not a closed development ecosystem. It's quite open. But it's a "do-acracy". Things get done because people do them... those doers, in turn, become maintainers. It's not at all difficult to submit a patch or a bug report, or even verify a reported bug on developer.blender.org . I've done all three and I'm not even a developer.
The barrier to entry isn't high at all. They only thing they don't do there is allow for random feature requests. If the feature request has a patch attached to it, they're more than happy to put it into the queue for review... but requests without someone to actually do them are better suited off the tracker.
* @Jason van Gumster * Well, that link sends you to developer.blender.org , where your bug can and most likely will be shot down if it's not up to standards. As I said, the developers put too much burden on the end user, many of which with little to no knowledge of software development processes.
* @bruno.souza.lino * So post a bug report that developers can re-produce. I don't see the problem with requiring good bug reports. If the error can't be consistently reproduced, then it can't be fixed because there's no way to track down the root problem.
That's not software development, that's basic troubleshooting... something that most artists are absolutely familiar with.
* @Jason van Gumster * Most artists also want a system that gives them assurance that their concerns are being heard. The current system is a gamble as the OP has pointed. I mostly see people working around bugs with add-ons or something else instead of reporting bugs. I wonder why.
* @bruno.souza.lino * Any other system is going to be just as much of a gamble. The proportion of general users to developers is enormous.
A large reason why most artists work around bugs rather than report them is because that's how they've been trained with proprietary software which doesn't give a fraction of the visibility into the development process that we have with open source software (and Blender specifically).