Severe lag in updating UI

Get answers to your questions about color grading, editing and finishing with DaVinci Resolve.
  • Author
  • Message
Offline

timoarnall

  • Posts: 9
  • Joined: Mon Jul 09, 2018 4:34 pm
  • Real Name: Timo Arnall

Severe lag in updating UI

PostMon Oct 15, 2018 1:53 pm

Hi there, we're experiencing a problem we can't resolve internally. Every edit change we make to our shared PostgreSQL project involves watching a spinning beachball for about 5 seconds.

Here's a gif to illustrate how frustrating the problem is:

Image

You can see there is about a 5.5 second delay on each UI click/drag/change.

This problem happens across 3 different iMacs (2018 models) and an iMac Pro (10 core) using DaVinci Resolve Studio 15.1.1.005.

All our machines are connected to a QNAP TVS-1282T3 Thunderbolt 12-Bay Network Attached NAS, with 28Tb of storage, an i7 processor and 32Gb of Ram. The connection is via Thunderbolt 3, which is super fast for everything else except UI changes :shock:

Any thoughts?
Offline
User avatar

roger.magnusson

  • Posts: 3354
  • Joined: Wed Sep 23, 2015 4:58 pm

Re: Severe lag in updating UI

PostMon Oct 15, 2018 6:30 pm

You can disable Live Save and enable Project Backups instead. Then every operation won't need to talk to the NAS. Not a remedy for the issue, but perhaps a usable workaround.
Offline
User avatar

Igor Riđanović

  • Posts: 1596
  • Joined: Thu Jul 02, 2015 5:11 am
  • Location: Los Angeles, Calif.

Re: Severe lag in updating UI

PostMon Oct 15, 2018 8:08 pm

Ping your server from one of the affected clients and see what kind of latency you get. It may be a general network issue.

Edited:

Sorry, I didn't see the Thunderbolt part. No ping there.
www.metafide.com - DaVinci Resolve™ Apps
Offline

timoarnall

  • Posts: 9
  • Joined: Mon Jul 09, 2018 4:34 pm
  • Real Name: Timo Arnall

Re: Severe lag in updating UI

PostTue Oct 16, 2018 10:18 am

roger.magnusson wrote:You can disable Live Save and enable Project Backups instead.


I don't think we can disable live save on a PostgreSQL database project, or am I missing something?
Offline

timoarnall

  • Posts: 9
  • Joined: Mon Jul 09, 2018 4:34 pm
  • Real Name: Timo Arnall

Re: Severe lag in updating UI

PostTue Oct 16, 2018 3:45 pm

We've just tried vacuuming the database, and nothing has changed. Any help appreciated!
Offline
User avatar

Igor Riđanović

  • Posts: 1596
  • Joined: Thu Jul 02, 2015 5:11 am
  • Location: Los Angeles, Calif.

Re: Severe lag in updating UI

PostTue Oct 16, 2018 5:53 pm

PostgreSQL comes with pgbench tool used to benchmark the DB performance. I've only used it a couple of times. I'm not 100% certain if it will report latency issues, but it's worth a try.
www.metafide.com - DaVinci Resolve™ Apps
Online

Peter Chamberlain

Blackmagic Design

  • Posts: 13874
  • Joined: Wed Aug 22, 2012 7:08 am

Re: Severe lag in updating UI

PostTue Oct 16, 2018 9:58 pm

Try a test with a clip on the local disk.
DaVinci Resolve Product Manager
Offline
User avatar

roger.magnusson

  • Posts: 3354
  • Joined: Wed Sep 23, 2015 4:58 pm

Re: Severe lag in updating UI

PostTue Oct 16, 2018 11:26 pm

timoarnall wrote:I don't think we can disable live save on a PostgreSQL database project, or am I missing something?

Normally you can, but not if you have enabled collaboration on the project.
Offline

timoarnall

  • Posts: 9
  • Joined: Mon Jul 09, 2018 4:34 pm
  • Real Name: Timo Arnall

Re: Severe lag in updating UI

PostFri Oct 19, 2018 10:17 am

roger.magnusson wrote: Normally you can, but not if you have enabled collaboration on the project.


Yep this is a collaborative project between three edit stations.
Offline
User avatar

waltervolpatto

  • Posts: 10502
  • Joined: Thu Feb 07, 2013 5:07 pm
  • Location: 1146 North Las Palmas Ave. Hollywood, California 90038 USA

Re: Severe lag in updating UI

PostFri Oct 19, 2018 3:33 pm

timoarnall wrote:
roger.magnusson wrote: Normally you can, but not if you have enabled collaboration on the project.


Yep this is a collaborative project between three edit stations.


I saw sticky edit when collaboration is on but it was on 14.3.1. I haven't tried I 15 yet.
W10-19043.1645- Supermicro MB C9X299-PGF - RAM 128GB CPU i9-10980XE 16c 4.3GHz (Oc) Water cooled
Decklink Studio 4K (12.3)
Resolve 18.5.1 / fusion studio 18
GPU 3090ti drivers 512.59 studio
Offline

timoarnall

  • Posts: 9
  • Joined: Mon Jul 09, 2018 4:34 pm
  • Real Name: Timo Arnall

Re: Severe lag in updating UI

PostFri Nov 09, 2018 10:38 am

Peter Chamberlain wrote:Try a test with a clip on the local disk.


Just tried a timeline inside the collaborative project with clips on the local iMac SSD, and we're experiencing the same 6 second lag.

I also tried a non-collaboration project on the Thunderbold NAS, and there is no lag, so this seems to narrow it down to a lag in updating the database? Anything we can do about this?
Offline
User avatar

Glenn Venghaus

  • Posts: 1358
  • Joined: Wed Jan 01, 2014 9:56 pm
  • Location: Amsterdam , The Netherlands

Re: Severe lag in updating UI

PostSat Nov 10, 2018 9:48 am

You dont specify the location of your database. Do you install/host it also on the nas ?
Nas is typicaly under equiped and has super small cpu / mem just for storage processing. And if its on the same nas as your media they will compete for bandwith as well.
Seems clearly an issue with either the network connection to the shared database or the shared database itself is having problems either with pure resources (cpu/memory) or its own storage.
You need to do some digging. Pgbench can help

Edit , just realised you qnap does have a beefy cpu and mem. So you can ignore that comment.

A quick thing you could to to see if its not an issue with the db itself (some hidden corruption ) and not the instance is to create a new empty db and copy the project over there and see if problem persists. That way you take at least one item of the checklist
Beatstep & APC-40 Resolve Edition Controllers https://posttools.tachyon-consulting.com
Test Rig : 2xXeon (24c) | UNRAID KVM OSX VM's | 128GB | 5700XT | 40Gbe
Prod Rig : i9-7940X (14c) | OSX 10.15 | 64GB | 2xVega 56 | 40Gbe | Tb3 | V:Eizo | A:5.1RME
Offline

timoarnall

  • Posts: 9
  • Joined: Mon Jul 09, 2018 4:34 pm
  • Real Name: Timo Arnall

Re: Severe lag in updating UI

PostMon Nov 12, 2018 10:16 am

Glenn Venghaus wrote:You dont specify the location of your database. Do you install/host it also on the nas ?


Hi Glenn, yes the database is installed on the NAS too, with 32Gb Ram and an i7.

Good idea to make a new database and test out on that, will do that, thanks!
Offline
User avatar

Glenn Venghaus

  • Posts: 1358
  • Joined: Wed Jan 01, 2014 9:56 pm
  • Location: Amsterdam , The Netherlands

Re: Severe lag in updating UI

PostMon Nov 12, 2018 11:32 am

No probs.

If that does not help i re-suggest use pgbench to test and possibly tune your database.
It will easily show you if you have an issue in your network as well or if the db instance itself is sluggish.

A quick setup would be :
On your workstation (so you include the network) make sure you have a postgresql client installed , which includes pgbench.
Then
1. Log on to the instance with psql and create an empty database named eg benchmark
Code: Select all
    <path to client binaries>/psql --host=<ip of server> --port=5432 -U postgres
     pqsql# CREATE DATABASE benchmark;
     pgsql#\q

2. Run pgbench to initialise this test database
Code: Select all
<path to client binaries>/pgbench --host=<ip of server> --port=5432 -U postgres -i -s 50 benchmark

3. Run a simple base benchmark . This example runs with 10 simulated (pgbench)clients , 2 threads and 10000 transactions.
Code: Select all
 <path to client binaries>/pgbench  --host=<ip of server> --port=5432 -U postgres -c 10 -j 2 -t 10000  benchmark


example output from a laptop calling a small mac mini installed shared database over wifi (so network worst case test, but can work comfortable on resolve on my laptop even with this):
Code: Select all
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 50
query mode: simple
number of clients: 10
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 100000/100000
latency average: 24.220 ms
tps = 412.889766 (including connections establishing)
tps = 412.908226 (excluding connections establishing)


now running it on the db server (macmini) itself so you can see the network effect even more clearly:
Code: Select all
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 50
query mode: simple
number of clients: 10
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 100000/100000
latency average: 0.000 ms
tps = 3263.303174 (including connections establishing)
tps = 3263.790093 (excluding connections establishing)


You can even make the network more involved (and so enhance any issues if there) in the test by adding the -C option , which will force a reconnection on "every" transaction (see latency average shooting up and tps down and specialy the diff in tps between including and excluding connection establishing).
Code: Select all
 <path to client binaries>/pgbench  --host=<ip of server> --port=5432 -U postgres -c 10 -j 2 -t 10000  -C benchmark

starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 50
query mode: simple
number of clients: 10
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 100000/100000
latency average: 75.181 ms
tps = 133.011475 (including connections establishing)
tps = 156.401047 (excluding connections establishing)

and here as a baseline from a 1GB wired connected workstation (with extended network option -C ) :
Code: Select all
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 50
query mode: simple
number of clients: 10
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 100000/100000
latency average: 35.431 ms
tps = 282.235230 (including connections establishing)
tps = 337.755780 (excluding connections establishing)


So lets say network is not an issue. A common cause for slow DB is a very/too small configured shared memory buffer. A common default is 128MB. Can still be enough pending usage , just play with it.
The configuration is in postgresql.conf and the parameter you are looking for is shared_buffers.
Do your pgbench test , adjust it, restart postgresql and rerun the pgbench test to see if it changes positively. Make small to medium adjustments and retest until you are happy.
Also double-check max_connections (number of maximum simultaneous db connections) but if you have only a few workstations it is likely ok (have set mine to 100 just to be sure, but have another (non resolve) app that is using up to 50 connections at once)

edit : extra remark regarding network. Make sure that you always use the "ip adress" of the database server in your resolve connection configs and not hostnames. This will avoid issues and delays in the hostname resolve process (dns), which can be a hidden problem i have see a lot that can lead to massive delays on each network request.

edit2: just to prove my own point , i upped my own shared_buffers (which was set back to default after some maintenance i did in the past and aparantly never noticed) from 128MB to 512MB and my local server tps went up from
Code: Select all
from before :
tps = 3263.303174 (including connections establishing)
tps = 3263.790093 (excluding connections establishing)
to after :
tps = 4443.717255 (including connections establishing)
tps = 4444.516070 (excluding connections establishing)


rem: my numbers are relatively low compared to possible on that hardware as the database is part of a realtime synchronisng cluster updating a second db. So that will reduce the numbers a lot, but the idea is the same. Just check on your local machine the relative numbers as discussed above Also the macmini is hosting another virtual machine. Just goes to show you dont need that much for a proper postgresql db.
Last edited by Glenn Venghaus on Mon Nov 12, 2018 3:23 pm, edited 1 time in total.
Beatstep & APC-40 Resolve Edition Controllers https://posttools.tachyon-consulting.com
Test Rig : 2xXeon (24c) | UNRAID KVM OSX VM's | 128GB | 5700XT | 40Gbe
Prod Rig : i9-7940X (14c) | OSX 10.15 | 64GB | 2xVega 56 | 40Gbe | Tb3 | V:Eizo | A:5.1RME
Offline

timoarnall

  • Posts: 9
  • Joined: Mon Jul 09, 2018 4:34 pm
  • Real Name: Timo Arnall

Re: Severe lag in updating UI

PostMon Nov 12, 2018 3:22 pm

Thanks so much Glenn, super useful.
Offline
User avatar

Igor Riđanović

  • Posts: 1596
  • Joined: Thu Jul 02, 2015 5:11 am
  • Location: Los Angeles, Calif.

Re: Severe lag in updating UI

PostMon Nov 12, 2018 9:04 pm

Glenn,

I have observed that the TPS value is not directly prortionate to shared_buffers value. In some cases larger buffer yields lower TPS. How could that make sense?

What do you think is sufficient TPS value for a single seat of Resolve?
www.metafide.com - DaVinci Resolve™ Apps
Offline
User avatar

Glenn Venghaus

  • Posts: 1358
  • Joined: Wed Jan 01, 2014 9:56 pm
  • Location: Amsterdam , The Netherlands

Re: Severe lag in updating UI

PostMon Nov 12, 2018 11:21 pm

Igor Riđanović wrote:Glenn,

What do you think is sufficient TPS value for a single seat of Resolve?


Impossible to quantify as such as there is no direct relationship between a simple test pgbench transaction and a resolve transaction.
Its just to do relative measurements and troubleshooting based on that. Also to see how to squeeze the max out of your system and measure the effect of parameter tuning effords. And as you have seen some parameter changes can flip the effect if you cross a threshold. Db tuning is an art and more is not always better. Too large shared buffers can actualy slowdown due to management overhead .
Also this is like any sort of cache mainly for read and not write . For write look into checkpoint segments tuning.
But sofar my experience is that for a single seat you dont need much power at all for a postgresql db in normal mode. In live safe mode its important to minimise any sort of networking latency issues as the individual transactions are negligeable from a db perspective and could even run on an raspberry pi. Also make sure the db is hosted on ssd so you have very low io waits as eventualy for writes every write has to be written to the filesystem.
Where you benefit from more db resources is in resolve startup , in full project saves, loads ,exports ,and imports cycles. These can seriously slow down on an underpowered db. And of course when you start multiseat you need to have configured enough resources (shared buffers, proper checkpoint segments tuning and fast io subsystem ) to handle simultaneous requests.

Recommeded reading starting point : https://wiki.postgresql.org/wiki/Tuning ... SQL_Server
Beatstep & APC-40 Resolve Edition Controllers https://posttools.tachyon-consulting.com
Test Rig : 2xXeon (24c) | UNRAID KVM OSX VM's | 128GB | 5700XT | 40Gbe
Prod Rig : i9-7940X (14c) | OSX 10.15 | 64GB | 2xVega 56 | 40Gbe | Tb3 | V:Eizo | A:5.1RME

Return to DaVinci Resolve

Who is online

Users browsing this forum: Baidu [Spider], Bing [Bot], cobra427, Google [Bot], panos_mts, Peter Chamberlain and 147 guests