Plugin manager for EG

[topix]

Hello community,

i'm currently working on a plugin manager for EG. Well, again. I already started a few month again and then stopped working on it, but now i resumed. On GitHub there is an issue for it where have been a little discussion about it.

The way i have implemented it is the following:
When the plugin manager (or optionally on EG start) is called, it get a list from a repository (for example: eg.net/plugins.php?myegversion=0.5.1&includeexperimentel=false), parses the list and shows the plugins in a GUI. From there the user can install/uninstall/upgrade the plugins.
The repository creates the plugin list from an database. I manually filled a database for testing. Cause i have not much experience with html and php, i weren't able to create a system for plugin sumitting and managing. My idea was to integrate it with the forum. So the registered users could submit a plugin and moderators then approve the plugin for integrating it in the repository. After that the user can themself remove the plugin or mark it as obsolete (when it is replaced by another plugin). But that's just my vision

There where concerns about security and the way of implementation. See the discussion on GitHub.

Now i hope to get some more feedback with this post and maybe asking for users with skills to write eventually the server side (the repository management).


Regards,
topix

[krambriw]

Hi Topix,

See here my worries & notes

viewtopic.php?f=12&t=9068&p=42886#p42886

[kgschlosser]

from reading the discussion on github it is being over complicated for no reason.

this should be set up using a simple download of a plugin using urllib. and the list of available plugins should be contained in a text file that cal also be download via urllib. if you want to utilize the tree in the add plugin dialog then great. there are not enough plugins to have to worry about doing a sql database. and if you keep it simple and use the security protocols that are currently working for the website. then leave them be and make it so that only downloads can take place, in all honest opinion someone needs to give a glance at any plugin that is made and see if the thing even will install into eg before it gets added to the list.

if there were thousands of plugins i would say do something more complicated. but until then there really is no need for it. an app can be written for whomever the "plugin moderator" is that will open the egplugin file and pull all the necessary data out of it for things like the icon and description and give a set of check boxes the moderator can use to set compatibility flags and the utility will add the url location to the egplugin file in the event someone is downloading it.


now this isn't Microsoft or Google and i don't for see someone targeting the EventGhost server for "hacking" now if some script kiddie happens along and someone left the door open. that's a whole other matter. and i will tell you this. the mysql connector for python sucks. it's horribly slow. so don't go that route. and a plugin repository should be located on some other server besides the core one anyways. there are tons of them out there. hell even the use of a public svn repo server would work and not load up on bandwidth usage on the eg server

[topix]

from reading the discussion on github it is being over complicated for no reason.

Yes, that was the the first version. I found a project with a plugin manager written in python and adopted it for EG. Currently i'm reducing the features and integrating it more to EG. As blackwind already mentioned, it would be nice to have it in the AddPluginDialog as a second tab. That way a user adding a plugin will directly be shown that there could be more plugins available.

[kgschlosser]

in all honesty because i do know that the original plan was to divert away from built in plugins all together. I would make it as a single feature. and put some kind of identifier telling the user if it has already been downloaded. but if the user wants to use a plugin if it is not already downloaded it will download it and then add it to the tree. if it has already been downloaded it will add it to the tree.


personally i don't like having plugins kicking around if i am not currently running it. and in my opinion if it gets removed from the tree it needs to be removed from the computer. but that being said not everyone shares the same feeling on that one.

bit as a side not. i do recall the use of the egplugin extension. and it is only a zip file. how about running the plugin from the zip kind of like how eg runs python from a zip. that would make it cleaner and less of a folder structure. and the reason for removing the plugin after it has been removed because in all reality how long would it take to download 10k? even on dialup that's only a couple of seconds. LOL. and it would be nice to trim the fat from eg except the core features like the script editor. (i really don't see a need for python command because of the editor) and enable disable jump. things of that nature. everything else should be a plugin that can be downloaded. keep core EG as only the items that control the core
functions. and there needs to be a better classification system for plugins as well. the standart "input" "external" needs to be a little more refined. that way you can add say an advanced search feature in the future.

control devices:
remotes.
mice.
keyboard.
.......
external equipment:
sound.
video.

software:
HTPC.
video players.
music players.
desktop.


and so on and so forth

the big thing is that people that download EG think that what is in the plugins list are the only things available. I do not recall seeing some sore of notice on the homepage or on the download page letting people know to search the forums for plugins. there is a lot more then meets the eye. and when the list starts to populate there needs to be some kind of keyword list that can be used from the egplugin file that can be used for searching and the classification setup for what group a plugin belongs to should allow for multiple groups.

[Sem;colon]

@kgschlosser: Didn't you say not over complicating?

I kinda said everything on GitHub already that I wanted to say.

I do like the idea of a plugin repository and in my opinion it should work without having some kind of moderator for it.
We have difficulties getting things done like a wiki, a forum, submitting to the code base etc.
I'm sceptical that we will get someone that is checking / approving plugins in time... But it would be an easy thing to add afterwards

My approach would be to focus on the core of the plugin manager first and add fancy things like categorisation afterwards.
We need some ground to start with.

[kgschlosser]

agreed. i was brain storming. not things that have to be added now. but thought should be given to additions. and to make whatever kind of plugin repo easily add to. that's more or less what i was trying to get at. easier to code in a way that would allow for expansion without having to rewrite the whole thing. Just sayin

[Sem;colon]

OK, I see.
I like the idea of using urllib, but in any case we need some central and online available place for all the plugins.
Does anyone have an Idea where to host this and how it should look like?

[kgschlosser]

well i keep on telling everyone. I have more then enough storage. and plenty of speed. and it's free. use it until something else is figured out. if the thought is using an svn then i will install an svn server. personally i would use http requests

[pearbear]

it should work without having some kind of moderator for it.
...
I'm sceptical that we will get someone that is checking / approving plugins in time.

As I see it there are two different actions that need to be handled:

• Add plugin to the manager.
• Pick up new versions of plugins that are already in the manager as they are released.

I agree that the most automation that can be added to this process the better. In a perfect world there would be someone reviewing every contributed plugin before it's added but that's a lot of work to expect volunteers to do in a timely manner. This would be a really big job for the initial population of the database but also ongoing as ideally each plugin version release should be checked also. A malicious actor could always make a simple benevolent 1.0.0 version of a plugin and then add the malicious code in the 1.0.1 release. Realistically I think it's best to just make sure that it's clearly stated which plugins are community contributed and add a warning that that they are to be used at your own risk. The community will have to be responsible for reporting any abuse of the system.

I do like the idea of using topics in the Plugin Support forum section to submit plugins to the manager since it builds on the current system. However, I think that will be a bit tricky to automate. It would require the plugin topics to use a fairly standardized format.

I think it would be a good idea if the plugin manager provides all released versions of the plugin, so that the user can select which version of the plugin they want to install and they are able to roll back to a previous version, rather than only having the latest version available in the manager. The reason is that new bugs or incompatibilities may be introduced in the plugin release. For example, if EG switches to Python 3 then the required changes in plugins may break backwards compatibility with older EG versions but some users may not be able to update their EG version due to using unmaintained plugins that are not compatible with the new EG version.

My closest experience with this sort of thing is the Arduino Library Manager. The way it works is the library author submits an issue to the Arduino IDE GitHub repository issue tracker requesting their library to be added to the manager with a link to the library repository. The library has to be hosted on GitHub, it has to have the correct folder structure, it has to have a correctly formatted metadata file, and it needs to have a git tag or release for each version. One of the Arduino developers has to check the library's repository to make sure it is correctly set up, then adds the repository URL in the Library Manager database. Even though the requirements are clearly documented, frequently there is an issue with the library repository and this requires some back and forth between the Arduino developer and the library author to get things straightened out. So just adding each library to the manager can be a significant amount of work for the person responsible for maintaining the database with that system. It seems like the submission process could be automated so that the library author would submit the library using a form and then would get a notification if the operation failed. Whenever the author tags/releases a new version of the library it is automatically picked up by the Arduino Library Manager within an hour so that part is completely automated. The downside is that this requires the library repository to be hosted on GitHub(though it should be possible to add support for similar hosts).

[Sem;colon]

I like the idea of an online form to submit plugins; just a file upload box that allows .egplugin files and some required fields for the metadata (maybe we dont even need them, as most of the info is in the .egplugin file already)
By this form the plugin will then be uploaded to a database or a file share, whatever will be the backend.
We would need some kind of user management for it though.

@kgschlosser: I don't know about the others, but I'd have no objections if you like to host the repository.

[krambriw]

@kgschlosser: I don't know about the others, but I'd have no objections if you like to host the repository.

My opinion is that such a repository MUSTbe in the cloud and more than 2 admins must have full access in case of accidents. For my small part, I have already today my contributed plugins in Google Drive. 15 GB is for free and will most likely be more than enough for all plugins (One of my largest, the RFXtrx, is just 68kB as an .egplugin compressed file).

[topix]

2016-11-07 20_17_13-Options.png (8.31 KiB) Viewed 5942 times

Testversion (Updated 08.11.2016)
Source code

The online database for this test is on a server i rent from a hoster.
I haven't yet searched the forum for plugins to add them to the list.

If you have problems, suggestions, wishes, remarks, etc... let's talk about it There is at least one thing to do: make a backup before plugin upgrade if something fails. And i'm sure there are more...

This is not the official way to go. It's just my version. And it's only the EG part. The online part for adding plugins to the repo is missing, so manually adding it through phpMyAdmin (or something similar) is needed.

[kgschlosser]

are you able to do the plugin upgrade on the fly without the need to restart EG??

[topix]

are you able to do the plugin upgrade on the fly without the need to restart EG??

Instalation works on the fly. Upgrading is what i'm now working on.

Edit:
Updated the setup file of my previous post.
It seems to be working, but for now i tested it only with "DesktopRemote". The plugin directory will be deleted befor installing the new version. So, if there are some files in use, i think it won't work.