Page 1 of 1

Tips for being a good beta tester

PostPosted: Mon Jul 16, 2018 7:12 pm
by mikedkelly
I see bits and pieces of this on different threads, but I thought it might be a good idea to summarize some reminders of how to best help the development team prepare their product for final release. I do not work for Blackmagic and these are my own opinions, so I encourage you to add, correct, and opine as you wish.

1. Beta-testing software helps both of us; Blackmagic gets a bunch of people finding bugs virtually for free, allowing them to release a better product for less cost, and we not only get to practice using the new features in advance, but we have a voice in what gets fixed and changed sooner rather than later.

2. Adding dozens of new features to an existing product of this size creates hundreds if not thousands of new workflow paths. It is simply not possible in any reasonable timeframe to create automated tests to cover every use case. Therefore, things are going to regress, break, crash, etc. It is the reality of software development and the result of us as end-users demanding major new versions of software every year instead of every decade.

3. It is not financially possible for Blackmagic to hire enough developers to fix every bug in the software. Therefore, they use an agile process where the product owner prioritizes the bugs and the developers fix them in iterations based on their capacity to build functionality in a given timebox. You see the results of these iterations as we get a new beta version every 2-3 weeks. Most software companies release their final production version based on the rate of incoming new bugs, not the number of open bugs. Thus the adage "Software is not released, it escapes."

4. The priority for fixing bugs is not based on reported displeasure, it is typically based on severity and the reported frequency of the issue. So, adding comments like “I can’t believe this doesn’t work” or “I’m going back to X software” has no impact and is not helpful. As a rule, leave all emotion out of your post. Just state the issue, your setup, how to reproduce, and logs if possible.

5. Responding to a thread with “I have this problem as well” is generally a good thing. It lets them know that it is not just an edge case with someone on an unusual hardware configuration or upgrade scenario. It is even better if you can respond with additional information like how your computer is similar or differs from the original poster, or narrowing down the issue with more specifics like “I can get it to work if I do X, but it always fails if I do Y.”

6. Personally, I prefer a bit of detail in the thread subject. “Resolve 15 broken” or “List of beta 6 issues” requires that I open the thread to see what is happening. As a former product owner, when I am looking at and prioritizing hundreds of issues, seeing a list of meaningful subject lines is much easier to work with than having to open every one to remember what something was referring to.

7. We have heard this a million times, but finally, don’t install a beta product in a mission critical environment. Telling Blackmagic how many hours you lost or what deadlines you missed on a customer’s project because of a beta bug is unprofessional and just reckless. If you don’t have a test machine that is separate from your production work environment, please strongly consider not being a beta tester.

My apologies if this has all been said in previous threads. If so, it will make its way off the page in a day or so. It’s also not a personal bash on anyone. If anything, it is a good reminder to me on how to post effectively.

Re: Tips for being a good beta tester

PostPosted: Mon Jul 16, 2018 9:43 pm
by Peter Fleming
A very good post and we should all make note of it.

Re: Tips for being a good beta tester

PostPosted: Tue Jul 17, 2018 5:00 am
by MikeRochefort
4. The priority for fixing bugs is not based on reported displeasure, it is typically based on severity and the reported frequency of the issue. So, adding comments like “I can’t believe this doesn’t work” or “I’m going back to X software” has no impact and is not helpful. As a rule, leave all emotion out of your post. Just state the issue, your setup, how to reproduce, and logs if possible.


I can’t stress this enough. The amount of this that happens across the web on software forums is crazy. I get it, we artists and technicians love our tools, and get frustrated when they don’t meet our expectations. But collectively as customers, I think we all need to take a step back from time to time and really try to provide as much help as we can to developers if we really want the product to get better.

I would also recommend using searchable tags, or tags for devs to quickly lock on to. For example, I use the metric of “[B#] [Tab] [OS] Subject Line”. So if Im having a problem on Beta 5 with Fairlight on my Linux workstation:

[B5] [Fairlight] [Linux] Audio mixing not remembering state

If a bug or problem get solved, add a [SOLVED] tag to the beginning and make a note of it in the first and last posts.

Cheer,
Mike

Re: Tips for being a good beta tester

PostPosted: Tue Jul 17, 2018 7:41 am
by Paul Draper
Nice, totally agree.

Re: Tips for being a good beta tester

PostPosted: Tue Jul 17, 2018 12:17 pm
by Rajiv Mudgal
This should be pinned at the top

Re: Tips for being a good beta tester

PostPosted: Tue Jul 17, 2018 4:28 pm
by Jean Claude
Hi,

A very simple thing to do and I do not know how many times asked by Peter, Dwaine and the whole BMD team:

Fill out your signature with your OS, Version, etc ...

All indications that will help to target the environment. Do not forget that BMD works with Windows, MAC and Linux ... it's an exponential job to manage 3 environments at the same time.

And to have it practice: in case of incident: post a link with a LOG and if possible screenshots: anything that can save time for the BMD team.

Be clear and precise in the description of an incident.

I find that the BMD team has the patience of an angel to ask each time the same information:
Before you post, please review ....
viewtopic.php?f=21&t=69837

;)

Re: Tips for being a good beta tester

PostPosted: Tue Jul 17, 2018 5:10 pm
by Mike Le Fevre
Here Here Totally concur and very well said

Re: Tips for being a good beta tester

PostPosted: Tue Jul 17, 2018 9:51 pm
by Mel Matsuoka
mikedkelly wrote:7. We have heard this a million times, but finally, don’t install a beta product in a mission critical environment. Telling Blackmagic how many hours you lost or what deadlines you missed on a customer’s project because of a beta bug is unprofessional and just reckless. If you don’t have a test machine that is separate from your production work environment, please strongly consider not being a beta tester.


Unfortunately ever since they made Gmail a public app, Google has really negatively changed the way civilians interpretation of what the term "Beta" really means. Nowadays, many people think "Beta" simply means "early access", as if you're being allowed elite access to something before all the rest of the peons get it.

I have ZERO sympathy for people who install Beta software, then complain on a public forum that they've lost all their work and getting angry at the software developer because of it. Just like I have ZERO sympathy for anyone who crosses a busy street without looking both ways, and then finds themselves flattened by an oncoming bus.

Re: Tips for being a good beta tester

PostPosted: Tue Jul 17, 2018 10:46 pm
by sacherjj
So if I'm understanding it is totally useless to post something like:

This isn't crashing enough!!!! I going back to Premiere. ;)

Re: Tips for being a good beta tester

PostPosted: Tue Jul 17, 2018 11:04 pm
by Tom Early
mikedkelly wrote:7. We have heard this a million times, but finally, don’t install a beta product in a mission critical environment.


I always like to adopt a 'what's the worst that can happen?' approach myself :D

Re: Tips for being a good beta tester

PostPosted: Wed Jul 18, 2018 1:52 pm
by Jason Tackaberry
Ok, let me play devil's advocate for a moment:
  • When you visit Resolve's product page the title says DaVinci Resolve 15
  • The product page tells you all about what's in Resolve 15
  • A download link for Resolve 15 Beta is prominently shown
  • The download for the free version of Resolve 14 is nowhere to be found on that page
  • It's actually not clear at all how to download Resolve 14 Free unless you know where to look (buried on the Support page)
  • There is no "Beta software! Use at your own risk!" warning
  • This product page has all the trappings of a Google style "early preview but pretty damn stable" type of beta

BMD is clearly very intentionally directing users of the free version to beta releases. The stable free version is completely buried, and so it's not entirely unsurprisingly that users will end up using the beta.

I do understand the value of the free version to both BMD and the community. It's a great opportunity for the community and BMD to create a symbiotic relationship. But rather than these users being fully informed and opting into a risky experience, they're really being treated as cannon fodder for Studio users. Consequently, some of these users will begin using this prominently featured product in good faith only to encounter serious issues due to beta quality.

Re: Tips for being a good beta tester

PostPosted: Wed Jul 18, 2018 4:48 pm
by Jean Claude
Jason Tackaberry wrote:Ok, let me play devil's advocate for a moment:
  • When you visit Resolve's product page the title says DaVinci Resolve 15
  • The product page tells you all about what's in Resolve 15
  • A download link for Resolve 15 Beta is prominently shown
  • The download for the free version of Resolve 14 is nowhere to be found on that page
  • It's actually not clear at all how to download Resolve 14 Free unless you know where to look (buried on the Support page)
  • There is no "Beta software! Use at your own risk!" warning
  • This product page has all the trappings of a Google style "early preview but pretty damn stable" type of beta

BMD is clearly very intentionally directing users of the free version to beta releases. The stable free version is completely buried, and so it's not entirely unsurprisingly that users will end up using the beta.

I do understand the value of the free version to both BMD and the community. It's a great opportunity for the community and BMD to create a symbiotic relationship. But rather than these users being fully informed and opting into a risky experience, they're really being treated as cannon fodder for Studio users. Consequently, some of these users will begin using this prominently featured product in good faith only to encounter serious issues due to beta quality.


Not quite agree with what is said :oops: :
here the page is "DaVinci Resolve 15 Beta"

"beta software! Use at your own risk!" Warning :

Everyone knows that a beta verion is a version under development: therefore not a release, so not definitive, so subject to evolution, so under development, therefore not suitable for production because not all test cases are not yet all developed, tested and validated according to the development plan and the validation test campaigns. So also check that there is no regression from one Beta version to another. And again and again until to obtain Release... :)

[FREE users] they are really treated like fodder for Studio users :oops: :
That the users respect in the first place this: (and you too)

Before you post, please review ....
viewtopic.php?f=21&t=69837

It is not normal to play riddles:
But which OS? Which version?
What material?

The simplest thing is to fulfill one's signature: ALL will gain time.

After: everyone can or not return an answer according to their skills and knowledge in an environment or others.

Do not forget that there is a WINDOWS, MAC and LINUX version (no matter the order) :)

Re: Tips for being a good beta tester

PostPosted: Wed Jul 18, 2018 6:07 pm
by Hendrik Proosa
Ain't no-one saying on BMD home page to go dig in user forum before touching beta version. Not every Resolve user, especially new users, reads this forum, and there is no soft or hard suggestion to do that before downloading and installing Resolve nowhere on Resolve product page, nor download registration form.

Everyone also knows that beta version should be in state of feature freeze, with main objective on bugfixing and stability. Far from it with Resolve and whether it is good or bad... it depends. I'd argue that a casual editor or whoever looking for new fancy software does not know squat about what beta means in this context anyway.

If you pay peanuts you get monkeys works the other way too. Free software means everyone is free to whine in this forum or wherever because they are first class users and have paid what was asked for.

Re: Tips for being a good beta tester

PostPosted: Wed Jul 18, 2018 6:54 pm
by Tom Early
Jason Tackaberry wrote:[*] There is no "Beta software! Use at your own risk!" warning


everyone knows Beta means 'use at your own risk', it's just common sense, as is backing up your projects regularly. I have no sympathy for anyone who cannot grasp these simple concepts.

Re: Tips for being a good beta tester

PostPosted: Wed Jul 18, 2018 7:18 pm
by Jason Tackaberry
Tom Early wrote:I have no sympathy for anyone who cannot grasp these simple concepts.
I might take a similar fuck-you stance with these users if the free non-beta version was actually clearly downloadable. As it is right now, I do have a bit of sympathy.

Re: Tips for being a good beta tester

PostPosted: Wed Jul 18, 2018 10:23 pm
by wutongdegugeyouxiang
Resolve has an automatic backup project feature. I set up to back up a project every 10 minutes, which is useful to prevent sudden damage or unexpected situations. It is recommended to set this up before making the project.

Davinci Reslove 15.b6 | CPU: i7 6700 | GPU: GTX 750 Ti | Windows 10 pro

Re: Tips for being a good beta tester

PostPosted: Wed Jul 18, 2018 11:50 pm
by Tom Early
wutongdegugeyouxiang wrote:Resolve has an automatic backup project feature. I set up to back up a project every 10 minutes, which is useful to prevent sudden damage or unexpected situations. It is recommended to set this up before making the project.

Davinci Reslove 15.b6 | CPU: i7 6700 | GPU: GTX 750 Ti | Windows 10 pro


doesn't have an automatic database backup feature though. That's something you need to do manually (and you need to do it regularly to be safe). Maybe it could be automated via scripting now though?

Re: Tips for being a good beta tester

PostPosted: Thu Jul 19, 2018 12:22 am
by Dan Sherman
Hendrik Proosa wrote:.
I'd argue that a casual editor or whoever looking for new fancy software does not know squat about what beta means in this context anyway.


Then they shouldn't be allowed to use a computer. What is it with society and wanting to dumb everything down for the laziest among us. I mean seriously, it takes 5 seconds to look up the definition of a word.

Jason Tackaberry wrote:
Tom Early wrote:I have no sympathy for anyone who cannot grasp these simple concepts.
I might take a similar fuck-you stance with these users if the free non-beta version was actually clearly downloadable. As it is right now, I do have a bit of sympathy.


all they had to do was search.
if you Google "davinci resolve dowload" the support page is the 3rd result. All they have to do is go to the page and scroll down the left column till they find the version they want.
https://www.blackmagicdesign.com/support/

I have no symphony for people who want something for free, and can't even be bothered to use their brains for a few moments to make sure it meets their needs.

Re: Tips for being a good beta tester

PostPosted: Thu Jul 19, 2018 12:32 am
by Peter Fleming
Pretty sad that so many have lost the plot on this posts intended purpose!!!!

Re: Tips for being a good beta tester

PostPosted: Thu Jul 19, 2018 5:45 am
by wutongdegugeyouxiang
Tom Early wrote:
wutongdegugeyouxiang wrote:Resolve has an automatic backup project feature. I set up to back up a project every 10 minutes, which is useful to prevent sudden damage or unexpected situations. It is recommended to set this up before making the project.

Davinci Reslove 15.b6 | CPU: i7 6700 | GPU: GTX 750 Ti | Windows 10 pro


doesn't have an automatic database backup feature though. That's something you need to do manually (and you need to do it regularly to be safe). Maybe it could be automated via scripting now though?


The path and database of the project backup are not in one location. Even if the database is damaged, it will not destroy the project. This update for Resolve is still very good.

Resolve's database is open and accessible from the resource manager. If the database is damaged, the project can still be restored by manually copying it to the new database. I have encountered this kind of thing once, but it quickly recovered.

Re: Tips for being a good beta tester

PostPosted: Thu Jul 19, 2018 6:31 am
by Hendrik Proosa
Peter Fleming wrote:Pretty sad that so many have lost the plot on this posts intended purpose!!!!

The first step in becoming a good beta tester is acknowledging that you are a beta tester in the firstplace. And this is where more and more users are flopping due to reasons perfectly clearly outlined by Jason Tackaberry. Now the question is what can BMD do about it, not whether people are dumbducks or whether they are to be let near computer. Because they are and they are, there is nothing anyone can do about that anymore.

A bit another topic but Blender user forum blenderartists.org has a pinned post called "How to Be a Positive Member of the Blender Community (don’t be a fanboy)" and something like this could be useful here too, especially with lots of new users coming in.

Re: Tips for being a good beta tester

PostPosted: Thu Jul 19, 2018 11:39 am
by MikeRochefort
Hendrik Proosa wrote:
A bit another topic but Blender user forum blenderartists.org has a pinned post called "How to Be a Positive Member of the Blender Community (don’t be a fanboy)" and something like this could be useful here too, especially with lots of new users coming in.


One of the nice things that the We Suck Less forum does is that when you create a new post in the bug tracker it has a bunch of preformatted text to guide you into a proper report. As it is also powered by phpBB I think it could be possible to implement that for just the Beta forum. A majority of posts at that point could be much more structured and provide more concise information.

Cheers,
Mike

Re: Tips for being a good beta tester

PostPosted: Thu Jul 19, 2018 4:43 pm
by Jean Claude
Come on !! I will give an extra layer :)

Tips for being a good beta tester:
1) know your subject
2) When one is subjected to a test case: do not hesitate to open the most recent reference manual at first in doubt
3) Are we talking about an existing feature that is evolving? : we have a track if we took care to read the manual!
4) Damn! it's new: what do I do ????
- I call the developers? => not very productive if it is always and always the case ... we become a handicap ...
- I pass and let the friends (the other beta testers) get by? In this case: Why be beta tester?
- I try to get by without knowing what to do and say anything? What is the risk? we'll tell you: do you really know the product you're testing? The best is to go take a bucket and a shovel and go play in your sand trap at the bottom of his office ...

In summary: as there is not one-tenth of speakers in this forum who have not even taken the time to learn (without offense): it is order to start with that! :roll:

Re: Tips for being a good beta tester

PostPosted: Fri Jul 20, 2018 6:58 am
by Hendrik Proosa
It has been said before, BMD should put up a proper public bug tracker. Reps have said they have a private tracker, but why not open it up for public? Bug reports are read by product managers and devs anyway, in my mind it is much easier to do it in an organized fashion without duplication. Writing stuff in random form into forum where bugs, issues, workflow questions and cat videos are mixed into one slurry soup is pretty bad for any reporting, beta or not. And it is totally fubar for users to track reported bugs in forum format because there is no reasonable way (categories, versions, status etc) to look for bugs already submitted. Random forum search with keywords is not it. Bug tracker also allows separating actual bug reports from help requests and other general topics. First ones are useful for devs, others are not.

Re: Tips for being a good beta tester

PostPosted: Fri Jul 20, 2018 8:33 pm
by Adrian Niwa
imho
biggest problem here is lack of possibility to install both versions on one machine.
So you have alpha v14
And beta v15.

You can work on an important project on v14 on different database and test stuff out on v15 separately.

As far as I'm working on a single machine I would have to set some sort of virtual machine to test v15 now, which would be pretty ridiculous.

Now you have to decide if you want to stay in the shadow of the past using v14, or maybe check v15 if it's stable and look through new features (which are pretty interesting btw).
So after you decide to check v15, you create one project with it, you're happy with it and you just crossed the line :) You don't want to go back anymore.
And then all the crap happens.

Two versions at one time on one machine would be a big "misunderstandings saver".

Cheers!

Re: Tips for being a good beta tester

PostPosted: Sat Jul 21, 2018 8:50 am
by Gabriele Gelfo
today there is no difference between a beta and final version because, for example, Adobe products have lot of bugs and issues also if they are final, I find updates every two days.
I consider all programs I use a working in progress, also my Apple Watch os has bugs.
Fortunately thanks the world wide web we can get updates constantly day and night.
BMD adds every months new features to Resolve, it means that the program is a wip, not finished yet and this progression will never end.
This fantastic, I love how they work because we can’t learn c++ programming but we can make artistic things in different ways using their software-hardware creations.
We ask and they listen.
eGpu is another example on how BMD listen his customers.

regards