Access the GPU without going through an X server

[dcommander]

There are spurious rumors that this either already is possible or will be possible soon with the nVidia drivers, by using EGL, but it is unclear exactly how (the Kronos EGL headers still seem to indicate that Xlib is required when using EGL on Un*x.) As soon as it is possible to do this, it would be a great enhancement for VirtualGL, since it would eliminate the need for a running X server on the server machine. I already know basically how to make such a system work in VirtualGL, because Sun used to have a proprietary API (GLP) that allowed us to accomplish the same thing on SPARC. Even as early as 2007, we identified EGL as a possible replacement for GLP, but Linux driver support was only available for it recently, and even where it is available, EGL still seems to be tied to X11 on Un*x systems. It is assumed that, eventually, that will have to change in order to support Wayland.

[tonyhb]

This would be awesome

[dcommander]

This functionality is indeed available in the latest nVidia driver, but I don't have it fully working yet. I can access the GPU device through EGL without an X server, create a Pbuffer, and (seemingly) render something to it, but I can't make glReadPixels() function properly, and I'm a little fuzzy on how double buffering and stereo can be implemented, as it seems like EGL doesn't support double buffered or stereo Pbuffers. Emulating double buffering and stereo using multiple single-buffered Pbuffers is certainly possible, but it would greatly increase the complexity of VirtualGL. Waiting for feedback from nVidia.

[dcommander]

After discussing at length with nVidia, it appears that there are a couple of issues blocking this:

ISSUE:

• Inability to select a specific GPU device using a device name
  Currently, you can enumerate all of the GPUs in the system (and obtain an EGLDevice handle for each one) by using the EGL_EXT_device_enumeration extension, and you can query each GPU by using the EGL_EXT_device_query extension, and you can obtain an EGLDisplay from one of those EGLDevices by using the EGL_EXT_platform_base and EGL_EXT_platform_device extensions. However, there is no way to obtain any sort of system-wide device name that is unique to a particular EGLDevice. VirtualGL would use this device name in the VGL_DISPLAY environment variable. The idea is that, if EGL is available, VirtualGL will use it by default, and the default value of VGL_DISPLAY will be "{EGL device name for GPU device 0}". Otherwise, if EGL isn't available, then GLX will be the default, and the default value of VGL_DISPLAY will be ":0.0". If the user specifies a particular X display as VGL_DISPLAY, then this would have the effect of forcing the use of GLX mode, even if EGL mode is supported. And setting VGL_DISPLAY to a particular EGL device name would cause all of the 3D rendering to occur on that device.

SOLUTION:

• nVidia is in the process of implementing the EGL_EXT_device_drm extension, which would allow for selecting a device based on its DRM filename (/dev//dri/card/{n}), and since this is a cross-platform extension, it would allow for selecting non-nVidia GPUs as well.

ISSUE:

• Lack of support for double-buffered and stereo Pbuffers
  EGL currently supports double buffering only for Window drawables. Pbuffers and Pixmaps are single-buffered. This is a problem for VirtualGL, because as it is currently implemented, VirtualGL assumes a 1:1 mapping between windows on the 2D X server and Pbuffers on the 3D X server. It relies on this 1:1 mapping, because it allows VGL to easily create Pbuffers with visual attributes that match the attributes that the application requests for a particular window. The most straightforward path to EGL support would be if VirtualGL could create EGLSurfaces that inherently support double buffering and quad buffering, so VGL could pass through SwapBuffers() and glDrawBuffer()/glReadBuffer() calls to the EGLSurface corresponding to a particular window. It doesn't appear that this is ever going to be possible with EGL, however.

POSSIBLE SOLUTIONS:

• Emulating double buffering and stereo using multiple Pbuffers
  Minimally extremely difficult and very prone to error. The problem is that this would require that VirtualGL swap contexts behind the scenes, whenever the application switches the drawing or read buffer. There would no longer be a 1:1 correspondence between OpenGL contexts maintained by VIrtualGL and GLXContext handles that VirtualGL returns to the application. At first glance, it might seem possible to maintain some sort of internal structure in VirtualGL (VGLContext, for instance), whereby each instance of the structure contains the context handles for all of the Pbuffers in question, and a pointer to this structure could be passed back to the application. Normally, VirtualGL just passes back the GLXContext handle from the 3D X server to the application when the application calls glXCreate[New]Context(), so the application is storing the GLXContext on behalf of VirtualGL. Conceivably, VirtualGL could pass back an opaque VGLContext handle to the application instead. My spidey sense tells me that I've already been down that road, and there were some issues associated with that, but I can't remember what they were (these were things that I dealt with very early on in the development of VirtualGL, so perhaps the issues no longer exist.) That aside, however, the show-stopping issue with this approach is whether VirtualGL could straightforwardly copy the relevant properties of one Pbuffer context to the other Pbuffer context whenever the application changes the drawing/read buffer. There doesn't seem to be an EGL equivalent of glXCopyContext(). We would ideally want to limit the touch points of this feature so as to minimize the risk of regression to the existing GLX-based implementation, and changing the internal meaning of GLXContext concerns me. I'd be much more comfortable simply passing an EGLContext back to the application when it requests a GLXContext. Furthermore, this proposal would no longer have a 1:1 mapping between Window handles on the 2D X server and GLXDrawable handles on the 3D X server, although I think it might be possible to maintain a mapping of multiple EGLSurfaces to one Window by using the existing VirtualDrawable::OGLDrawable class.
• Emulating double buffering and stereo using FBOs
  This would allow a single EGLContext to be mapped to a single GLXContext, which eliminates some of the concerns above. We could maintain a 1:1 mapping between Window handles on the 2D X server and GLXDrawable handles on the 3D X server by using a "dummy" 1x1 Pbuffer and attaching FBOs to it. All of that could be abstracted within the existing VGLDrawable::OGLDrawable class. Swapping buffers could be implemented simply by swapping the FBO attachments. There are issues with this approach, however:
  • Emulation of aux and accumulation buffers. Aux buffers would be easy to emulate using FBOs, since they act as completely independent draw/read buffers, and the application is tasked with transferring pixels into and out of them. Accum buffers would be more difficult. They would require that VirtualGL emulate the glAccum() function, which is likely to be tricky at best. Aux and accum buffers are now an obsolete feature (as of OpenGL 3.1), so it might make sense to say "if you want those features, you have to use the legacy mode-- i.e. GLX mode-- in VirtualGL."
  • Handling stenciling, depth buffers, and other standard OpenGL visual features. These would all have to be maintained as separate FBOs, all attached to the drawable on the 3D X server. The FBOs would have to be created to match the properties of a particular EGLFBConfig (primarily alpha channel, depth bits, and stencil bits.) This isn't impossible, but it does add complexity to the solution, since VirtualGL would have to remember which attributes were used to create a particular EGLFBConfig (within the body of glXChooseVisual() or glXChooseFBConfig()), so that it could create and attach the FBOs appropriately within the body of glXMake[Context]Current().
  • How to handle applications that already do FBO rendering on their own. Care would have to be taken to ensure that the application's FBO attachments do not conflict with the attachments that VirtualGL is making behind the scenes. This would probably require that VirtualGL interpose the functions related to FBO creation and binding so that it could avoid using attachments that the application requests. There is a high likelihood of introducing application-specific compatibility issues here.
  • How to handle the fact that FBOs do not persist across contexts. Although not the common case, there certainly are applications that use multiple OpenGL contexts with the same drawable. VirtualGL would have to detect this case within glXMake[Context]Current() and transfer pixels between the FBOs used by the old context and the FBOs used by the new context, if the two contexts are bound to the same drawable.
• Emulating double buffering and stereo using the EGL multiview extension
  There exist two extensions (EGL_EXT_multiview_window and EXT_multiview_draw_buffer) that, at least architecturally, would seem to solve the FBO issues described above, since multi-view buffers are attached to a particular drawable, not a particular context, and the multi-view buffers inherent the default visual attributes of the drawable. However, it would be necessary for the driver vendor to implement an equivalent EGL_EXT_multiview_pbuffer extension to allow multiview with Pbuffers and to implement EXT_multiview_draw_buffer using full OpenGL (as opposed to GLES.) I see this approach as being the most straightforward and as providing the lowest possibility of regression, but there is not a good sense that the driver vendors will be willing to implement the necessary extensions just to support us.

[dcommander]

Simple program to demonstrate OpenGL rendering without an X server:
git clone https://gist.github.com/dcommander/ee1247362201552b2532

[dcommander]

Popping the stack on this old thread, because I've started re-investigating how best to accomplish this, and I've been tinkering with some code over the past few days to explore what's now possible, since it's been two years since I last visited it. AFAICT (awaiting nVidia's confirmation), the situation is still the same with respect to EGL, which is that multi-view Pbuffers don't exist. That leaves us with the quandary of how to emulate these GLX features:

1. Double buffering. The lack of a multi-view Pbuffer EGL extension would require that we:
   1. emulate Pbuffers using FBOs, since double-buffered Pbuffers wouldn't exist. Currently VirtualGL emulates OpenGL windows using Pbuffers, but in the new implementation, it would have to emulate Pbuffers as well. We could probably still create a 1x1 dummy Pbuffer for each OpenGL window, which would at least allow us to maintain the 1:1 relationship between Drawable handles on the 2D X server and GLXDrawable handles on the 3D X server (or EGLSurfaces), but the actual structure of the emulated Pbuffer would be implemented with a "Drawable FBO" (and appropriate RBO attachments to emulate the back, stencil, and depth buffers.) This is problematic, since we'd be attempting to map a lower-level OpenGL feature to a higher-level GLX feature. The requirements would include, but would probably not be limited to:
      1. Interposing glReadBuffer(), glDrawBuffer(), glDrawBuffers(), glNamedFramebufferReadBuffer(), and glNamedFramebufferDrawBuffer() (VGL already interposes glDrawBuffer()) and redirecting GL_FRONT, GL_BACK, GL_FRONT_AND_BACK, etc. to the appropriate GL_COLOR_ATTACHMENTx target (in the case of GL_FRONT_AND_BACK, this would require calling down to glDrawBuffers().) Fortunately it appears as if it is an error to call glDrawBuffer() or glReadBuffer() with a target of GL_BACK/GL_FRONT/etc. whenever an FBO other than 0 is bound, so VirtualGL can similarly trigger an OpenGL error if those targets are used without the Drawable FBO being bound.
      2. Interposing glBindFramebuffer() in order to redirect Buffer 0 to the Drawable FBO.
      3. Interposing glGet*() in order to return values for GL_DOUBLEBUFFER, GL_DRAW_BUFFER, GL_DRAW_BUFFERi, GL_DRAW_FRAMEBUFFER_BINDING, GL_READ_FRAMEBUFFER_BINDING, GL_READ_BUFFER, and GL_RENDERBUFFER_BINDING that make sense from the application's point of view.
   2. emulate GLXFBConfigs somehow, since the GLXFBConfig or EGLConfig of the emulated Pbuffer would not represent its visual properties necessarily. This would likely require that VGL maintain a central table of internal FB configs; perform its own sorting algorithms within the body of glXChooseVisual(), glXChooseFBConfig(), and similar functions; and return its own internal structure pointers to the application when the application requests a GLXFBConfig. This is feasible, but it's difficult and fraught with potential compatibility issues.
   3. emulate GLX_PRESERVED_CONTENTS (Hopefully we don't need to? Otherwise, I have no clue), GLX_MAX_PBUFFER_WIDTH and GLX_MAX_PBUFFER_HEIGHT (could map to GL_MAX_FRAMEBUFFER_WIDTH and GL_MAX_FRAMEBUFFER_HEIGHT), and GLX_LARGEST_PBUFFER.
2. Quad-buffered stereo. If we have to use FBOs to emulate double-buffered Pbuffers, then this would be an easy addition, Otherwise, I don't mind relegating this feature to the GLX back end only. I'm trying to figure out the industry direction on stereographic 3D rendering in general, because at the moment, it doesn't even appear possible to use quad-buffered stereo in OpenGL without using GLX. Furthermore, the only VGL configuration that supports quad-buffered stereo is the VGL Transport with a Linux client that has stereo capabilities. That configuration is useful for accessing visualization supercomputers remotely across a LAN, so it's definitely something I want to continue supporting, but it doesn't necessarily need to be supported with X-server-less GPU access.
3. Aux. buffers. If we have to use FBOs to emulate double-buffered Pbuffers, then this would be an easy addition. Otherwise, this feature won't be available with X-server-less GPU access (aux. buffers were obsoleted in OpenGL 3.1 anyhow.)
4. Accumulation buffers. These can't be emulated with FBOs, so if we have to use FBOs to emulate Pbuffers, then support for accumulation buffers will simply not exist in VirtualGL anymore. Accumulation buffers were also obsoleted in OpenGL 3.1, but why do I have a sinking feeling that there are still some commercial applications out there that use them? I guess such applications would have to be stuck on VGL 2.5.x if the use of FBOs proves necessary.
5. Floating point pixels and other esoteric Pbuffer configurations that the nVidia drivers support. No idea even where to begin emulating such things using FBOs.
6. Texture-from-pixmap. There appears to be an EGL extension for this, but no idea whether it supports desktop OpenGL or just OpenGL ES.
7. Buffer swapping. If we have to emulate Pbuffers using FBOs, hopefully we can get away with simply swapping the color attachments.

Features that will likely have to be relegated to the legacy GLX back end only:

• glXSelectEvent(). If we have to use FBOs to emulate Pbuffers, then I'm not sure how to emulate this at all. (Bueller? Bueller?)
• GLX_EXT_import_context and indirect contexts in general. EGL has no concept of indirect contexts.
• GLX_NV_swap_group. If we have to use FBOs to emulate Pbuffers, then this extension may not be possible to emulate at all.

As you can see, this is already a potential compatibility minefield. It at least becomes a manageable minefield if we are able to retain the existing GLX Pbuffer back end and simply add an EGL Pbuffer back end to it (i.e. if a multi-view EGL Pbuffer extension is available.) That would leave open the possibility of reverting to the GLX Pbuffer back end if certain applications don't work with the EGL Pbuffer back end. However, since I can think of no sane way to use FBOs for the EGL back end without also using them for the GLX back end, if we're forced to use FBOs, essentially everything we currently know about VirtualGL's compatibility with commercial applications would have to be thrown out the window. Emulating Pbuffers with FBOs is so potentially disruptive to application compatibility that I would even entertain the notion of introducing a new interposer library just for the EGL back end, and retaining the existing interposers until the new back end can be shown to be as compatible (these new interposers could be selected in vglrun based on the value of VGL_DISPLAY.)

Maybe I'm being too paranoid, but in the 13 years I've been maintaining this project, I've literally seen every inadvisable thing that an application can possibly do with OpenGL or GLX. A lot of commercial OpenGL ISVs seem to have the philosophy that, as long as their application works on the specific platforms they support, it doesn't matter if the code is brittle, non-future-proof, or if it only works by accident because the display is local and the GPU is fast. Hence my general desire to not introduce potential compatibility problems into VirtualGL. The more we try to interpose the OpenGL API, the more problems we will potentially encounter, since that API changes a lot more frequently than GLX. There is unfortunately no inexpensive way to test a GLX/OpenGL implementation for conformance problems (accessing the Khronos comformance suites requires a $30,000 fee), and whereas some of the companies reselling VirtualGL in their own products have access to a variety of commercial applications for testing, I have no such access personally.

[dcommander]

Relabeling as "funding needed", since there is no way to pay for this project with the General Fund unless a multi-view Pbuffer extension for EGL materializes.