EventGhost

But I guess I may have to go down on the frequency of logging in the end.

Yes, and maybe distribute the report creation around the hour. If you have a report with numerous nbr of tables involved, a heave reading of the db occurs. Even if there is no writing to the db, a number of threads are started. Since we do not really know where the limit of concurrent running threads are (especially inside a VM), I think you can adjust the upper limit until we see the "old" problem coming back.

Since the Q goes up that high, I fear that there are some threads that are not terminating correctly (could be threads used for report creation).

But spreading the execution a bit would not harm...

It seems like something must be done, yeah. I have another VM that does some work related video conversion, which is a bit heavy on the CPU, and it seems like that's when it happens. Because just now it has gone up to the 45 and locked again. I'll change the report generation a bit and then we'll see.

Weird, I was working on the new setup (each report triggers the next after two minutes), and suddenly the queue just started to tick again.

And now it's down to 1! Well, I have made a system now with the two minutes delay for each of the five reports that are triggered every hour, and then finally triggers the FTP upload. I'll try that until tomorrow and see what happens.

Great, that will be interesting
Thanks!

Changing the report generation to 10 minutes, not 2. Two weeks worth of data takes a bit of time, it seems...

After the generation is done the que seems to fall again. Now I'm starting a new video conversion while running reports. And then I'm going to check in half an hour to see the damage!

My friend, you are really hitting the machine hard!

Yep, I am! I work hard, I play hard, I fu... Well, never mind! The queue is up to 41 now, it seems to keep growing even when I'm only running one report and video at the same time. I guess I'll have to increase the size of the queue to around 100! Would that be dangerous? The video conversion is only taking about 20 minutes, and I do them 3-4 times a day.

Would that be dangerous?

No it's not, the thing that can happen is that we eventually get that old exception back again ('Can't start thread...') due to limited resources. But otherwise, just try, it might be that 100 is still far from the upper limit. And if you get that out of resource problem, it could be that giving the VM more RAM could also help.

Up to 141 before the video conversion was done at 13.27, after around 27 minutes. Then it started to shrink rather slowly, bobbing up and down like 132-133-134-133-132-131 and so on.

I can try more memory since I have so much available. CPU use is around 60-70 when the queue is shrinking, but it's only using a flatline of 1,26 gig RAM, so I'm not sure if that can help.

But some days the video conversion goes almost non stop for a couple of hours. So maybe I have to put the queue at 600? It hasn't gone any lower than 75 after 15 minutes, so I guess this will take some time to get under control again!

Yep, and soon there will be another round coming up...

Maybe considering physically separating home automation from video amusements...I mean it is not possible to make a standard Volvo hitting 310 km/h even if downhill, longing for home and need for a toilet visit.

...or if possible, set the priority of your video VM to very low

But it's possible for it to reach 712.76 km/h in free fall! Which in my opinion is all a Volvo is useful for... I'd rather have a Koenigsegg! But seriously, this is work, not amusement. That means that when I'm at the cabin it has to be easily accessible from both home (when my wife works from home) and the cabin. I have fiber at home, bad DSL at the cabin, so sending to the cabin is fast, the other way is 1/50 of the speed... It's down to 20 now.

Edit: Almost sync posting (which is a lot like sync swimmning, except for that I don't wear a bathing suit, I have no idea what you're wearing...)! That could be possible, but I have the video stuff at normal, and so is the VM. Oh, and now it's down to 5! So now let's see what happens when there's another round of reports in five minutes, this time without the video conversion.

Reporting without the video conversion is no problem. So I guess I'll have to make a change in collecting data when I convert video.