ProcessWatcher

Questions and comments specific to a particular plugin should go here.
Sem;colon
Plugin Developer
Posts: 700
Joined: Sat Feb 18, 2012 10:51 am
Location: Germany

ProcessWatcher

Post by Sem;colon » Fri Jan 27, 2017 9:14 pm

Hi together,

I recently did some work on the ProcessWatcher plugin I'd like to share with you.
The ProcessWatcher plugin is a handy plugin, shipped with EventGhost, that monitors your running processes and triggers events when a new process starts or ends.

I noticed that this plugin consumes a lot of CPU power because it checks for new processes every 0.1 seconds, so 10 times per second!
I increased this value to every 0.5 s, which is still a lot (2 times per second, you will not notice any difference) but consumes 5 times less CPU cycles.

Additionally I changed it to include the pid in the payload of the event and added two new actions for scripting, that help you checking if a process is currently running or not.

Oh, and there was also a threading issue that it didn't terminate correctly.

Have fun!
Attachments
ProcessWatcher.egplugin
v1.1.0
(3.04 KiB) Downloaded 166 times
If you like my work, Image me a drink :wink:

jonib
Plugin Developer
Posts: 1344
Joined: Thu Mar 26, 2009 9:33 pm
Location: Sweden

Re: ProcessWatcher

Post by jonib » Sat Jan 28, 2017 6:08 am

Sem;colon wrote:I noticed that this plugin consumes a lot of CPU power because it checks for new processes every 0.1 seconds, so 10 times per second!
I increased this value to every 0.5 s, which is still a lot (2 times per second, you will not notice any difference) but consumes 5 times less CPU cycles.
I haven't looked at the code, but changing the time from 0.1s to 0.5s might not seem like big deal, but there are processes that live only a very short time and checking only two times a second there is a big chance to miss them. Even the 0.1s seems like it will miss some.

I don't know how likely it is that it would miss a process for which you want an event for just something to consider.

Seems like there should be a better way to keep track of when processes start/stop via some WinAPI, quick search found Example: Receiving Event Notifications Through WMI (Example in C++)
Might be my next project (event though I haven't finished any of my current projects :roll: but new projects are more fun then old :twisted: )

jonib
XBMC2 plugin to control XBMC. If you want to flatter me Image

Sem;colon
Plugin Developer
Posts: 700
Joined: Sat Feb 18, 2012 10:51 am
Location: Germany

Re: ProcessWatcher

Post by Sem;colon » Sat Jan 28, 2017 10:33 am

I agree that there is still room for improvement, but I didn't find anything "easy" to get around the process polling - and as you wrote, you may miss processes with 0.1s sleep as well.
As long as polling is used, we can't be sure that every process is catched. I assume that the processes that run > 0.5 seconds are the more interesting, but of course there could be use cases for even catching the smaller ones.
I took a look at http://pythonhosted.org/psutil/ which provides quiet some nice features to monitor processes, but our use case here seems not to be covered.

The high cpu load of this plugin was always a pain point for me
If you like my work, Image me a drink :wink:

jonib
Plugin Developer
Posts: 1344
Joined: Thu Mar 26, 2009 9:33 pm
Location: Sweden

Re: ProcessWatcher

Post by jonib » Sat Jan 28, 2017 6:41 pm

So I was able to make a Python implementation of the WMI example, and it works but unfortunately it's still polling but on the Windows side.
I was hoping there was some reliability advantage with this code but it seems to perform about the same.

This code is for only process creation events, looks like there needs to be another thread for ending processes.

Code: Select all

    def ThreadLoop(self, stopThreadEvent):
        pythoncom.CoInitialize()
        try:
            c = wmi.WMI()
            raw_wql = "SELECT * FROM __InstanceCreationEvent WITHIN 0.5 WHERE TargetInstance ISA 'Win32_Process'"
            watcher = c.watch_for(raw_wql=raw_wql, wmi_class="Win32_Process")

            while not stopThreadEvent.isSet():
                try:
                    Process = watcher(1000)
                except wmi.x_wmi_timed_out:
                    pass
                else:
                    eg.TriggerEvent("Created." + Process.Name, prefix="Process", payload={"pid": Process.ProcessId})
        finally:
            pythoncom.CoUninitialize()
Fortunetly I didn't have to waste too much time on this. :D

jonib
XBMC2 plugin to control XBMC. If you want to flatter me Image

jonib
Plugin Developer
Posts: 1344
Joined: Thu Mar 26, 2009 9:33 pm
Location: Sweden

Re: ProcessWatcher

Post by jonib » Sun Jan 29, 2017 8:40 am

So I was disappointed yesterday that my WMI code didn't seem any better, but today I know better, because I got an idea maybe the WMI polling is a bit more efficient?

So I boosted the 1.1.0 version to 0.0001s polling rate so one of my processor cores get maxed.

Then I tried the same setting on my WMI code and it seems to use no more CPU power then when disabled. So much more efficient. :shock:

Edit: So the CPU usage test might not been as impressive as I first thought, the polling rate on the WMI code don't seem to match 0.0001s. When using both versions 1.1.0 version was always first in the EG log. doing more testing but the WMI code is definitely more efficient.

Edit2: The WMI code don't show the CPU usage in the EG process so it needs more testing if it actually is more efficient or not, working on it.

Edit3: Weeel, the WMI code actually uses more CPU power, it was just hidden in a system process so I missed it before. I think I'm finished with this for the time being.

So I guess I'll work on this a bit more today, yay.

jonib
XBMC2 plugin to control XBMC. If you want to flatter me Image

User avatar
kgschlosser
Site Admin
Posts: 5402
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: ProcessWatcher

Post by kgschlosser » Mon Jan 30, 2017 1:39 am

there has to be a way to hook CreateProcess to do this and I think that would be a more efficient way about it...That would be beyond my coding abilities at the current time.. but might be able to point ya in a direction that may help

who knows...


and reason why the WMI approach doesn't run at the thread polling rate is because the WMI takes up time to do things. and what you are seeing is a delayed response. meaning that a process can activate do what it needs to do then exit before you even get an indication that the process even started. so you would be at the mercy of WMI at that point
If you like the work I have been doing then feel free to Image

User avatar
kgschlosser
Site Admin
Posts: 5402
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: ProcessWatcher

Post by kgschlosser » Mon Jan 30, 2017 1:51 am

but maybe the solution is really simple.. just add a control to the plugins config panel to allow the user to change the polling interval.

even better would be an action to dynamically change the speed of the interval. because there may be some kind of an event that occurs that would cause the user to want to "ramp up" the polling speed because they may be looking for something to happen and then they would be able to "ramp down" the speed when it does.

just a thought

K
If you like the work I have been doing then feel free to Image

jonib
Plugin Developer
Posts: 1344
Joined: Thu Mar 26, 2009 9:33 pm
Location: Sweden

Re: ProcessWatcher

Post by jonib » Mon Jan 30, 2017 2:58 am

kgschlosser wrote:there has to be a way to hook CreateProcess to do this and I think that would be a more efficient way about it...
That is probable the most efficient way, but also most low-level and might not be easy from Python. Another way might be something called ETW don't know if it would be better then via WMI.
but maybe the solution is really simple.. just add a control to the plugins config panel to allow the user to change the polling interval.
I think that's the easiest improvement so lower powered machines can use a slower polling rate. On my machine this plugin with the default 0.1s polling uses way less the 1% CPU so I never noticed a problem.
even better would be an action to dynamically change the speed of the interval. because there may be some kind of an event that occurs that would cause the user to want to "ramp up" the polling speed because they may be looking for something to happen and then they would be able to "ramp down" the speed when it does.
Interesting, sound like a nice solution when needing to catch some short lived processes temporarily.

jonib
XBMC2 plugin to control XBMC. If you want to flatter me Image

User avatar
kgschlosser
Site Admin
Posts: 5402
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: ProcessWatcher

Post by kgschlosser » Mon Jan 30, 2017 5:47 am

want me to do up a change and submit it as a PR?? I am thinking something along the lines of having something to change the polling time.. and then a timer to change to back.. but to also add a default value in the plugin config... and if you don't want to use the timer to change it back set the timer to 0 and that will keep it at whatever you set in the action. until you restart EG or change it again using the action or in the plugin config.
If you like the work I have been doing then feel free to Image

User avatar
kgschlosser
Site Admin
Posts: 5402
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: ProcessWatcher

Post by kgschlosser » Mon Jan 30, 2017 7:07 am

i went ahead and did the modifications and sent them in for a PR. here is the file if you wanted to give it a test run
Attachments
__init__.py
(8 KiB) Downloaded 146 times
If you like the work I have been doing then feel free to Image

Sem;colon
Plugin Developer
Posts: 700
Joined: Sat Feb 18, 2012 10:51 am
Location: Germany

Re: ProcessWatcher

Post by Sem;colon » Mon Jan 30, 2017 9:49 am

Hi K,

what's with all the other modifications I did?
Did you even look at the plugin I uploaded??

Would you mind using the version I uploaded and and add your modifications?

thx!
If you like my work, Image me a drink :wink:

User avatar
kgschlosser
Site Admin
Posts: 5402
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: ProcessWatcher

Post by kgschlosser » Mon Jan 30, 2017 1:54 pm

@Sem;colon

My bad i didn't see the fact you added extra things like the pic.. I saw the timing thing.. yeah yeah... I'll do it now and send it on up.. sorry my bad on that..

Have you tried the bit on the closing of the dialogs that i coded up based on your idea??? I didn't know if ya saw that or not
If you like the work I have been doing then feel free to Image

User avatar
kgschlosser
Site Admin
Posts: 5402
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: ProcessWatcher

Post by kgschlosser » Mon Jan 30, 2017 3:57 pm

Hey I have a test run for ya to make sure i didn't booger up your additions... I had to change out some things like the use of eg.TranslatableStrings . and I also removed the use of Iterkeys() because if we ever change to python 3 it will be less work to have to convert this stuff later... may as well get into the habit of doing it now. and i combined the repeat code for the triggering of an event into a single method. because i had to add some more things to it.. but let me know what you think...


Oh btw... I also added another feature.. I think you are going to like it...

I will hold off on doing a commit for these changes until i hear back from you. and I will also make sure your name gets put in the changelog as well..
If you like the work I have been doing then feel free to Image

jonib
Plugin Developer
Posts: 1344
Joined: Thu Mar 26, 2009 9:33 pm
Location: Sweden

Re: ProcessWatcher

Post by jonib » Mon Jan 30, 2017 6:50 pm

I found this Deviare .DLL library that hooks into Windows APIs to get events.

And using this short test Python script:

Code: Select all

# -*- coding: utf-8 -*-
import win32com.client

class NktSpyMgrEvents(object):
    def OnProcessStarted(self, nktProcessAsPyIDispatch):
        nktProcess = win32com.client.Dispatch(nktProcessAsPyIDispatch)
        print("Started:", nktProcess.Name, nktProcess.Id, nktProcess.Path)

    def OnProcessTerminated(self, nktProcessAsPyIDispatch):
        nktProcess = win32com.client.Dispatch(nktProcessAsPyIDispatch)
        print("Ended:", nktProcess.Name, nktProcess.Id, nktProcess.Path)


if __name__ == "__main__":
    spyManager = win32com.client.DispatchWithEvents('DeviareCOM.NktSpyMgr', NktSpyMgrEvents)
    spyManager.Initialize()

    import ctypes
    MessageBox = ctypes.windll.user32.MessageBoxA
    MessageBox(None, 'Press OK to end the demo.', 'Deviare Python Demo', 0)

    import sys
    sys.exit()
I get process events without any polling (as I understand it). Unfortunately I haven't been able to get it to work in EventGhost yet (probably something stupid) so I haven't been able to see if it has any advantage, but it should not be able to miss any processes and that is my main concern with the default polling code.

jonib
XBMC2 plugin to control XBMC. If you want to flatter me Image

User avatar
kgschlosser
Site Admin
Posts: 5402
Joined: Fri Jun 05, 2015 5:43 am
Location: Rocky Mountains, Colorado USA

Re: ProcessWatcher

Post by kgschlosser » Mon Jan 30, 2017 7:13 pm

yeah I know i saw the same thing...

but this

Code: Select all

win32com.client.DispatchWithEvents('DeviareCOM.NktSpyMgr', NktSpyMgrEvents)
I believe it's hooking it's own device driver/dll because of the first parameter.

Now.. that being said... cFunctions is the hooking that has been made for eventghost. but there isn't anything in there that has been designed to help this. and I have hard a hard time trying to locate any realtime process monitoring as an example... but we have to know what to attach to first that will spit out messages when a process is created and terminated. now I think you have to do this using the kernel32 not sure tho.. again wayyyyy over my programming skills...well kinda.. I did modify and add all the extra power notifications and battery level ups battery level if the computer is plugged in to the wall or not.. monitor states.. all that kind of crap... but there is an api designed to spit out these events in the windows api and is not hooking or leveraging an api to get the information.. and I am not sure how to do it without the need to thread poll
If you like the work I have been doing then feel free to Image

Post Reply