[Feature request] Multiple keyboards and mice

[davidawesome02-backup]

Hey, this is just a question because I am not sure how to contact you otherwise, feel free to remove if it is too out of scope.

I mentioned what I originally wanted to do, before seeing this project started, in the gamescope repo, here. This is essentially what has already been done in part with bwrap, and specialized kwin scripts to place gamescope windows (unless I am misunderstanding).

My question boils down to if, and how, I should go about making multi-keyboard setups work, and generalize this kind of experience to more outside of gaming (and kde). Gamescope seamed like a good solution, similar to the reasoning to the mentioned splinux repo for my ideas, but here you seam to already have started on what I would want.

Should I attempt to follow my design goals for this project in upstream gamescope, and hope to then indirectly allow for multi keyboard support here?

I have not looked into this too much, and unfortunately don't have much experience yet, so I am still looking into what I should do, and how.

If you have any ideas, let me know, otherwise you can just remove this issue, as it may not directly relate to the project.

ValveSoftware/gamescope#1771

[wunnr]

Splinux, as you mentioned, does support assigning keyboards and mice to specific instances; I don't know how this is achieved, as since starting this project I've purposefully refrained from looking at its code beyond a simple skimming, because I didn't want to actually plagiarize any of it when making partydeck. At some point, when my life gets less busy, I want to ask the author if they are willing to have their implementation borrowed for PartyDeck for KB/M support.

I do try to keep up with updates to the software that I'm using, so if Gamescope/KWin/Bubblewrap/Proton ever get updates or features that improve the splitscreen/multiseat experience I would try to implement them ASAP.

In the meantime, if you wanted to contribute to PartyDeck, a good starting place would be looking into if other DEs like GNOME have robust scripting APIs that allow for the kind of thing that I'm already doing with the KWin script.

[davidawesome02-backup]

I will look more into how I can do this as a gamescope feature as of this moment, because doing so there, or possibly in bwrap or similar.

I was thinking of emulating the keyboard in libinput, and using gamescope (as previously stated), and may do this in a wrapper. The program splinux, I know you cant copy directly because GPLv3 is copyleft, but seeing as its still opensource, and the idea he / she used is novel, I will just explain it below, and may patch my self if I take the time to learn some rust.

All they do is use xtest api to send fake keyboard events from a non existent keyboard, and just bind to all keyboards specified so xorg or wayland or libinput or whatever cant access them at that point. It then sends the keys as it reads them directly back to the xtest api, exposed from a x11 crate.

If you want to add this, it would be cool, but I will most likely try to do mouse and keyboard input my self in gamescope, hopefully allowing it to be done without all this required for games that dont need it.

Also, for other DE's, is there a reason we cant use native x11 apis? I know wayland dose not have a great api for this unless we get the screen size, and place them relative to another window we spawn, but mby even bundling a basic tiling window manager, or returning similar would allow us to have a single window (gamescope) that contains the actual games inside said tiler.

Also... I have not checked this project much, and I see its still new, but do you currently have x11 and wayland support, or do you plan on just wayland (x11 only would work via xwayland)?

sorry if my rambling makes zero sense, feal free to let me know how best I should do this if you have any better ideas. I am a bit late to reply due to work and other things, sorry :P

[wunnr]

Using xtest to send inputs to each gamescope is a pretty clever idea. Doing the same thing in partydeck would require rewriting the app to be async so that one thread could be doing the input stuff; right now it just launches the instances through a basic shell command, and once the instances are launched the app is frozen until they're closed. Very doable, but I'm a rookie, so it would take a bit of time for me to learn how to do it.

It would be helpful if gamescope had flags for x and y window position; combined with the width and height flags it would give us the tiling effect without having to use a kwin script, making it DE agnostic. I don't know what would be required to add that functionality to gamescope, or if it would even work on Wayland.

Right now the KWin script seems to work equally well on the x11 and wayland versions of KDE, though most testing has been done on x11 (my steam deck is my daily driver right now, unfortunately)

Hopefully that answers your questions, lmk if you need any clarification or have additional ideas on this

[davidawesome02-backup]

Hey, so just published a few changes that could make gamescope work with specified keyboards rather than all of them in a way that should work with this. Please take a look if you have time as I think it would make a good addition to this project if it ever gets merged. If it dose not, allowing custom gamescope would also allow for this feature to be merged, and would be interesting. I know it can currently be done by modifying the PATH, but its up to you if you wish to use these features. My friend and I are looking into making a modification to add the new support into this project (he knows rust much better than I) but LMK if you see anything I should change :)

ValveSoftware/gamescope#1897

EDIT: forgot to mention, with wayland, we cant specify window positions directly, but it might be possible to have gamescope in a subgamescope window to specify the position, although this would remove the whole full screen single window they intended for it. Might get merged, but they may not care enough for that. Doing the actual positioning of the gamescope window (if single instance) is not possible as wayland has no api that officially dose that iirc.

[wunnr]

This looks neat! Will try this build myself when I get the time.

Can you pass through mice using --libinput-hold-dev? If so, does each gamescope instance get their own mouse cursor separate from the host's cursor? I see some stuff about mice in the PR but just want to be sure.

[davidawesome02-backup]

Yes actually! I made it so it renders a software composited cursor using their existing apis for what I can only assume to be the DRM backend, so both keyboards and mice work on the --libinput-hold-dev option as long as you specify the /dev/input/by-id/ path of said device. Default software compositing is done only in the case of you also passing --backend-disable-mouse as well, as without this option, you are expected to also use your main computers mouse, and I didn't want to have to check for if each specified device was a mouse, keyboard, or what type before enabling the software compositor.

LMK if you think there should be any other changes, and feal free to try it if you have time, my friend and I will see if we can figure out rust and patch in this into this project in a bit :)

[wunnr]

So if I'm understanding correctly, if I pass in a mouse to a gamescope window I should also be passing --backend-disable-mouse so that the host's mouse does not interfere with the gamescope session?

[wunnr]

Built the gamescope fork and did Did some preliminary testing, seems like it does work! There were a few quirks I ran into while capturing mouse/keyboard input, but I believe the PR thread is the better place to discuss that, not here. Tomorrow I'll start work on letting users add keyboards and mice from inside the partydeck app.

I'll also be changing the title of this issue to be more focused on kb/m specifically.

[davidawesome02-backup]

Hey, so I am still thinking of how I want to implement this, if I should use this project or make a slightly different one, and somewhat of a related thing I wanted to point out as possible changes I/you/we could also make (I am not sure if I am going to stick to this project entirely for my ideas, but I will share them, because you may be interested, and I may still try to commit them here, I don't know yet):

• Support for multiple devices per slot / session / window (keyboard and mouse games would require this)

• Support for tiling window manager instead of kwin scritps (Still looking into making this a gamescope patch, will get back to you on that, right now sway is known to work [x11 / wayland / has embedded mode], but the project I would prefer, dwm dose not [x11 only, no embedded mode]) *(this is needed for a later feature request)

• Support for split keyboard binding / mby controller too. (this might be complex for controllers, but for keyboards, I can add into my upstream gamescope patch a mask that gets applied to the keyboard keys before they are processed, and as long as we can use (idk what it is in rust) c's select, we can duplicate a fd so it is only locked by gamescopes, and dose not get detected by the default manager still (I think good?), otherwise we can in just this project (A lot of work) use uinput to emulate a new device sending masked keyboard data, to presplit the keyboard (would allow a window manager to take the rest of the keys or a different game to take each half).

• Support for running games just by specifying (in steam) the commandline splitscreen before any args, or splitscreen %command%, giving us a wrapper that would allow us to extract almost anything we need from the game runtime, rather than having to make our own launch scripts like you do right now. (would also allow for playing the game inside steam big picture mode (steamdeck's default). The only hard part from this would be that I believe that steamos's install only allows flatpaks to easily be installed (appimages, or raw executables require going to google 👍) and both these options would add another bwrap session to our data, but flatpak still seams to have that disabled for "reasons".

• Mby use brwap's --overlay feature to allow for all user files to be modified (edited files would appear in a directory we setup for them), rather than specifying specific paths in the config file (current setup as far as I can see). (also would allow running the game in what it thinks to be its normal launch location (no files copied still), and thus make games that may be location dependant work properly [I used to see this as a bug, not sure if its still a issue today].

The only problem I see if how to get goldberg working well, as it has to be in a specific place it looks like, but then again, have not read much of the code doing goldberg emulation yet.

sorry for the run on information, thinking of changes that could make this better, but I have to find time to implement :)

[davidawesome02-backup]

Here, not sure if you would want it, but I made the changes required for input filtering if they decide to add them: ValveSoftware/gamescope#1907

[wunnr]

@davidawesome02-backup Sorry for the late response.

Contributions to the project are always welcome, but if you want to make changes that you feel are outside the scope or goal of the main partydeck, feel free to make a fork! As for your suggestions, I'll try to address them point-by-point.

1. Support for multiple devices per instance is planned!

2. Tiling window manager support I'm not so sure about; each WM/DE has their own API for modifying window geometry, so supporting all of them would require learning all of the APIs or working with some universal x11/wayland api for doing that. If you get any more info on this I'd appreciate updates.

3. Splitting/masking keyboards by certain keys in the launcher seems a bit niche, so I won't be prioritizing it right now.

4. I could add support for launching through commandline: maybe the syntax could look something like ./partydeck-rs --game bl2proton --gamepads=/dev/input/event17,/dev/input/event20 --profiles=guest,profile1? One big problem is that the app's GUI mostly exists to select what devices (gamepads,kb/m) will be used for the sessions before launching, so if we launch through cmd there might need to be some other way of doing that.

5. Havent heard of bwrap overlays, but based on how you describe it it sounds like it might be useful. Will look into.

[wunnr]

Also, if your input holding support for gamescope doesn't get merged into upstream, I'd still like to utilize it in partydeck. Do you have a suggestion as to how best to distribute a binary of your gamescope fork? Ideally, the binary isn't included with the program, but rather gets downloaded automatically on first launch, the same way the other dependencies are currently handled right now.