Monday, November 22, 2010

A High-Level Overview of Grabs

One of the most common topics coming up in X input is grabs. This is a high-level overview of input device grabs and how they affect input event processing. Note that the examples here are simplified, grabs are probably the most complicated topic in input processing.

By default, X input events are delivered to all clients that registered for a specific event on a window. The basic premise of a grab is that it affects event delivery to deliver events exclusively to one client only.
There are three types of grabs, two classes of grabs, and two modes for a grab. The three types are:

  • active grabs

  • passive grabs

  • implicit passive grab



The two classes are core grabs and device grabs and the two modes are synchronous and asynchronous.
A grab comes in a combination of type + class + mode, so a grab may be an "active synchronous device grab".

Grab Types


Let's look at the difference between types first.

Active Grabs


An active grab is a direct result of a client request. The client requests "Grab this device and deliver its events exclusively to me". Future events are then delivered to this client until the client "ungrabs" the device again.
From a UI point-of-view, active grabs are commonly used when events must go to one specific application, regardless of the pointer or keyboard focus. One example are popup menus, where you want the next click to either activate a field or remove the window when clicking outside of the popup menu.

Passive Grabs


A passive grab is a promise by the X server to grab the device next time a button or key is pressed. The client essentially requests "grab the button when it is on window W and Alt and Shift are down". This request is stored and in the future when the pointer moves into window W, the specified modifiers are down and the user presses the button, the passive grab is activated (so now we have an "activated passive grab") and events are delivered exclusively to the client that requested the passive grab. Once the button or key is released, the passive grab is released again (and events are sent to all applicable clients). Unlike active grabs, passive grabs will activate repeatedly whenever the conditions are met. Ungrabbing a passive grab means to remove the conditions that trigger the activation of a passive grab.

XI2 added enter/leave and focus in/out passive grabs in addition to the already common button and key grabs. So a passive grab may activate when a pointer enters the window now.

Implicit Passive Grabs


These are a special type of passive grabs that are not under the direct influence of clients. Whenever a button press event is delivered normally to a client (i.e. no passive grab activated), this client gets an implicit passive grab on the device to ensure that future events, most notably the release event, is also delivered to the same client. Implicit passive grabs only come in synchronous mode, more on that later.


Grab Classes


In the core X protocol, only pointer and keyboard grabs are defined. With the addition of the X Input Extension, the device grabs were added as well. The only real difference is that with the latter a client explicitly specifies which device to grab. In the case of a core grab, the client simply says "grab the pointer" or "grab the keyboard", in the case of a device grab, the client must specify a valid device to grab. The class of a grab affects the events being sent to the client. A client with a core grab cannot receive X Input events from this device and vice versa.
With the addition of MPX and XI2 to the server, core grabs now work according to the ClientPointer principle. Since one of the requirements for MPX/XI2 was to not interfere with core applications, the behaviour of the two classes of grabs differs slightly.


  • A core grab is a promise that the device will only send events to the grabbing client, and that no other device may interact with the client while the grab is active.

  • A device grab is a promise that the device will only send events to the grabbing client but other devices may still send events to this client too.



The basic assumption here is that core applications do not know about the existence of multiple devices, but XI-aware applications do. Other than that, the two classes behave mostly identical.

Grab Modes


Grabs, with the exception of implicit passive grabs, may come as synchronous or asynchronous grabs. Asynchronous are simple: the event stream is processed by the X server and sent to the grabbing client immediately. During synchronous grabs, the server will queue up events internally and ask the client how to continue for each event. On a synchronous passive button grab for example, the client will receive the first event (the button press) but no more. It can then decide what to do and instruct the server how to proceed with the events. One common instruction is to "thaw" the device, i.e. release the event stream and continue in asynchronous mode. The other common instruction is to "replay" the event. This is a common operation for window managers which put a grab on mouse buttons as it allows them to reshuffle windows before the event is seen by an application. The list of actions is essentially this:


  1. Button press occurs on window W.

  2. Synchronised passive grab for window manager WM activates.

  3. WM receives button event.

  4. WM brings window W to the foreground.

  5. WM requests to "replay" the event.

  6. X server deactivates passive grab.

  7. The same button press event is now sent to W and the clients listening for button events on W.



Misc Other Info


If a client requests an active grab, you are at the mercy of this client. If the client decides to not ungrab (for whatever reason, usually a bug), then you will not be able to use the device on any other application. This is what's commonly referred to as "stuck grab". The only way to get out of this at the moment is to find the offending client and kill it. All grabs for a client are automatically terminated when a client exits.

A passive grab may be converted into an active grab. If a client already has a passive grab and requests an active grab on the same device, this active passive grab is converted to an active grab and thus does not terminate automatically anymore. The client then has to send an ungrab request to the server. This is a common operation for popup menus, a client has a synchronised passive grab that is converted into an asynchronous active grab once the passive grab activates. It only works passive → active though, an active grab cannot be converted into a passive grab.

All grabs are based on to a window and the event coordinates in the subsequent events are relative to that window's origin. During a grab, the pointer or keyboard is regarded as inside this window (even when outside the window's boundaries) and thus enter/leave and focus in/out events are sent as if the device had left the previous window. These event will have a flag set to notify other clients that the leave or focus out event was caused by a grab.

If you think all this is confusing, look at the code that implements this :)
Also note that this post skips on a number of details and contains some amount of handwaving and sticking fingers in your ears and going "lalalala".


[edit] fix typo, on synchronous button grabs the client receives only the first event.

Thursday, November 18, 2010

How to ignore configuration errors

Two common errors turning up in the log files are "(EE) ioctl EVIOCGNAME failed: Inappropriate ioctl for device" and "(--) SynPS/2 Synaptics TouchPad: no supported touchpad found".

One example for the former is:

[ 59.837] (II) config/udev: Adding input device ADS7846 Touchscreen (/dev/input/event3)
[ 59.837] (**) ADS7846 Touchscreen: Applying InputClass "default"
[ 59.841] (**) ADS7846 Touchscreen: Applying InputClass "evdev touchscreen "
[ 59.842] (**) ADS7846 Touchscreen: always reports core events
[ 59.842] (**) ADS7846 Touchscreen: Device: "/dev/input/event3"
[ 59.842] (--) ADS7846 Touchscreen: Found absolute axes
[ 59.842] (--) ADS7846 Touchscreen: Found x and y absolute axes
[ 59.842] (--) ADS7846 Touchscreen: Found absolute touchscreen
[ 59.843] (II) ADS7846 Touchscreen: Configuring as touchscreen
[ 59.843] (**) ADS7846 Touchscreen: YAxisMapping: buttons 4 and 5
[ 59.843] (**) ADS7846 Touchscreen: EmulateWheelButton: 4, EmulateWheelIner0
[ 59.845] (II) XINPUT: Adding extended input device "ADS7846 Touchscreen" ()
[ 59.851] (II) ADS7846 Touchscreen: initialized for absolute axes.

--------------------- divider added for clarity ----------------

[ 59.854] (II) config/udev: Adding input device ADS7846 Touchscreen (/dev/input/mouse0)
[ 59.854] (**) ADS7846 Touchscreen: Applying InputClass "default"
[ 59.855] (**) ADS7846 Touchscreen: always reports core events
[ 59.855] (**) ADS7846 Touchscreen: Device: "/dev/input/mouse0"
[ 59.855] (EE) ioctl EVIOCGNAME failed: Inappropriate ioctl for device
[ 59.856] (II) UnloadModule: "evdev"
[ 59.856] (EE) PreInit returned NULL for "ADS7846 Touchscreen"


One example for the latter:


[ 494.925] (II) config/udev: Adding input device SynPS/2 Synaptics TouchPad (/dev/input/event6)
[ 494.925] (**) SynPS/2 Synaptics TouchPad: Applying InputClass "evdev touchpad catchall"
[ 494.925] (**) SynPS/2 Synaptics TouchPad: Applying InputClass "touchpad catchall"
[ 494.925] (II) LoadModule: "synaptics"
[ 494.927] (II) Loading /usr/lib/xorg/modules/input/synaptics_drv.so
[ 494.927] (II) Module synaptics: vendor="X.Org Foundation"
[ 494.927] compiled for 1.9.0, module version = 1.3.99
[ 494.928] Module class: X.Org XInput Driver
[ 494.928] ABI class: X.Org XInput driver, version 11.0
[ 494.928] (II) Synaptics touchpad driver version 1.3.99
[ 494.928] (**) Option "Device" "/dev/input/event6"
[ 495.282] (--) SynPS/2 Synaptics TouchPad: x-axis range 1472 - 5682
[ 495.282] (--) SynPS/2 Synaptics TouchPad: y-axis range 1408 - 5076
[ 495.282] (--) SynPS/2 Synaptics TouchPad: pressure range 0 - 255
[ 495.282] (--) SynPS/2 Synaptics TouchPad: finger width range 0 - 0
[ 495.283] (--) SynPS/2 Synaptics TouchPad: buttons: left right
[ 495.283] (--) SynPS/2 Synaptics TouchPad: invalid finger width range. defaulting to 0 - 16
[ 495.522] (--) SynPS/2 Synaptics TouchPad: touchpad found
[ 495.522] (**) SynPS/2 Synaptics TouchPad: always reports core events
[ 495.682] (II) XINPUT: Adding extended input device "SynPS/2 Synaptics TouchPad" (type: TOUCHPAD)
[ 495.683] (**) SynPS/2 Synaptics TouchPad: (accel) MinSpeed is now constant deceleration 2.5
[ 495.683] (**) SynPS/2 Synaptics TouchPad: MaxSpeed is now 1.75
[ 495.683] (**) SynPS/2 Synaptics TouchPad: AccelFactor is now 0.036
[ 495.683] (**) SynPS/2 Synaptics TouchPad: (accel) keeping acceleration scheme 1
[ 495.683] (**) SynPS/2 Synaptics TouchPad: (accel) acceleration profile 1
[ 495.683] (**) SynPS/2 Synaptics TouchPad: (accel) acceleration factor: 2.000
[ 495.683] (**) SynPS/2 Synaptics TouchPad: (accel) acceleration threshold: 4
[ 495.842] (--) SynPS/2 Synaptics TouchPad: touchpad found

--------------------- divider added for clarity ----------------

[ 495.843] (II) config/udev: Adding input device SynPS/2 Synaptics TouchPad (/dev/input/mouse0)
[ 495.843] (**) SynPS/2 Synaptics TouchPad: Applying InputClass "touchpad catchall"
[ 495.844] (II) Synaptics touchpad driver version 1.3.99
[ 495.844] (**) Option "Device" "/dev/input/mouse0"
[ 495.922] (--) SynPS/2 Synaptics TouchPad: invalid x-axis range. defaulting to 1615 - 5685
[ 495.922] (--) SynPS/2 Synaptics TouchPad: invalid y-axis range. defaulting to 1729 - 4171
[ 495.922] (--) SynPS/2 Synaptics TouchPad: invalid pressure range. defaulting to 0 - 256
[ 495.922] (--) SynPS/2 Synaptics TouchPad: invalid finger width range. defaulting to 0 - 16
[ 495.943] (EE) Query no Synaptics: 6003C8
[ 495.943] (--) SynPS/2 Synaptics TouchPad: no supported touchpad found
[ 495.943] (EE) SynPS/2 Synaptics TouchPad Unable to query/initialize Synaptics hardware.
[ 495.944] (EE) PreInit returned NULL for "SynPS/2 Synaptics TouchPad"


In both cases, you'll notice that the device gets added twice. Once with a /dev/input/eventX device path, and once with a /dev/input/mouseX path. The first one succeeds and that results in your device being added to the X server's device list. The second one fails because both evdev and synaptics try to use an ioctl that is only available on event devices. The errors do not indicate a driver bug but a misconfiguration. The local configuration needs to be adjusted to avoid non-event devices being assigned to evdev or synaptics.

Look at the synaptics xorg.conf.d snippet we ship in Fedora:

Section "InputClass"
Identifier "touchpad catchall"
Driver "synaptics"
MatchIsTouchpad "on"
MatchDevicePath "/dev/input/event*"
EndSection


That last line is important. It only tries to assign the synaptics driver to event devices, not to mouse devices. Once you've added that line (or poked your distro maintainer to do it for you), the errors will go away. Alternatively, you could just ignore the errors and not file a bug for it :)

Note that the synaptics upstream snippet does not contain this line. Synaptics supports more than just Linux so we cannot add this upstream.

Monday, November 15, 2010

XKB mouse emulation removed from default keymap

XKB has a feature referred to as "MouseKeys" or in the case of xkeyboard-config "PointerKeys". Once enabled, the keypad controls the pointer movement and can be used to emulate button events. The default keymap hat the PointerKeys toggle on Shift+NumLock.

Like so many other things, it is a low level feature that has virtually zero desktop integration. The only way to find out whether it is on or not is to hit a key and see the mouse cursor move. That of course requires you to already know about the feature.

If you've accidentally enabled it the only thing you may notice is that your keypad stopped working. Worse, with a nasty bug we had in servers 1.7 and later (fixed now), the mouse buttons would get stuck when PointerKeys was used intermixed with physical mouse button presses. I've received a fair number of bug reports and other complaints about this feature that I finally decided to remove it from the default options. Sergey has pushed the patch to the xkeyboard-config repository over the weekend, so expect that in xkeyboard-config 2.1 and later, PointerKeys will need to be explicitly enabled.

You can do so with the XKB option "keypad:pointerkeys", or by ticking the "Toggle PointerKeys with Shift + NumLock" checkbox in your desktop keyboard configuration UI.

Like the changed zapping behaviour and the middle button emulation default (both of which have completely failed to bring the world to an end, despite claims to the contrary), this is only a changed default. The feature is still there and can be re-enabled on-the-fly or permanently if needed.

Thursday, November 4, 2010

Adding Reviewed-by tags to patches

I merge a lot of patches and (thankfully!) I get a lot of Reviewed-by, Acked-by, etc. comments on both my patches and others that I end up merging. Copying those is painful and quiet annoying. Somewhen last year, I stumbled across snipMate which allows you to add autocompletion snippets to vim. My $HOME/.vim/snippets/gitcommit file now looks like this:


snippet ack
Acked-by: ${1}
snippet rev
Reviewed-by: ${1}
snippet sob
Signed-off-by: ${1}
snippet tested
Tested-by: ${1}
snippet me
Peter Hutterer <peter.hutterer@...>


and the names of the common patch reviewers. So whenever I comment on a patch or I amend a commit after receiving reviews, just typing "rev<space>name alias<space>" is enough to add the Reviewed-by: line.

snipMate is quite powerful but so far I haven't used it much beyond the above. But even just that saved me numerous hours.

Thursday, October 28, 2010

XDS 2010 videos

Matt Dew announced that he put the videos from XDS 2010 on youtube. If you're interested about the various topics we talked about, peruse it at your convenience.

For those that don't know him, Matt has been tackling the challenge of collecting X-related documentation (see his talk) for a while now. Someone give that man a medal, or alternatively I guess a suitable beverage will do.

Saturday, October 2, 2010

Thoughts on Linux multitouch

Note: I am speaking for myself here, in my role as input maintainer for X.Org. The opinions here are my own and may not be the same as those of my employer, other X.Org developers or anyone else alive or not.

Two weeks ago, I was in Toulouse, France, at a multitouch workshop organised by Stèphane Chatty. After the workshop, in the same week was XDS. The workshop had a nice mix of people, Benjamin Tissoires whom I credit with kicking off much of the multitouch work with his evdev hacks, Pengfei, a PhD student of Stèphane, and Chase Douglas from Canonical, involved with their multitouch efforts. Ping Cheng from Wacom, Gowri Ries and Pascal Auriel from Stantum represented the hardware side. And Zeno Albisser and Denis Dzyubenko from Nokia for the toolkit side (Qt). We talked multitouch for two days and I think we got a lot done - not in code but in concepts and general design questions. The current state is essentially that both hardware and software guys are waiting on us, the X server, to integrate multitouch.

The workshop also happened shortly after the Ubuntu multitouch announcement and ended up being an excellent setting to discuss what was, is, and will be happening in regards to multitouch support on Linux. This post is a long one, take your time and don't just skim through. I will focus more on concepts and approaches, not on hard technical stuff. That is for a later post, mainly because I've been on vacation since and I have not yet caught up with new patches.

Multitouch basics


Let's start from the basics. Traditional input devices such as mice provide one physical device to control one virtual input point (the cursor or the keyboard focus). At it's very basic, multitouch is the ability to use multiple input points on a single physical device. This has some implications on technical issues but many more implications on the UI design. It is also the main reason why multi-pointer (which we support since server 1.7) is not a replacement for multitouch.

Multitouch comes in roughly two different forms: direct touch and what I call dependent touch. Direct touch is simple, a touch point directly relates to the position on the device, i.e. a touch in the upper right corner of the touch device will usually result in some action in the upper right corner of the screen. Dependent-touch devices are e.g. multi-touch aware touchpads but also devices like the Apple Magic Mouse. In their case, the touch is usually interpreted relative to the pointer position of that same device.

The usage of such devices differs vastly too. A direct-touch device will likely be used as such, with plenty of direct object manipulations at their respective position. This again has UI implications: a common design challenge for direct-touch user interfaces is to ensure that all objects are within reach of the user. By the way, from a purely technical point of view, the "touch" part of direct-touch is irrelevant.

A dependent-touch device has a stronger dependency on gestures; a simple touch on such a device doesn't necessarily have a meaning. Touch capabilities on these devices provides an augmentation of traditional input actions, similar to extra buttons or knobs. As usual, the world isn't black and white only and this is only a rough description with plenty of corner cases.

The lack of context



The hardest thing about multitouch is the difference between the data we get and the context the user assumes. Touch devices see touches appear and disappear but cannot (yet) identify appearing and disappearing touches and their context across time. I'll give you one example of the mismatch between the data we get and the context a user may assume: In a drawing application, the following events happen: First, a touch appears on the colour selector and selects the colour red, then the touch disappears. Second, a touch appears on the colour selector and selects the colour green, then the touch disappears. Then two touchpoints appear on and move across the canvas.

Possible scenarios that cause the above datastream are:
  • The user has erroneously selected red, then corrected to colour green, now is painting with two fingers in green.

  • The user has selected red with one finger, green with another finger and now wants two paint with two different colours.

  • The user has erroneously selected red with one finger, corrected to green and now wants to paint in green and the colour the other finger already had assigned to.

  • Two users selected two different colours and now paint simultaneously.



Which one it is is impossible to know from the data stream. Hence the difficulty of creating good multitouch interfaces, the data we get only carries a fragment of the information that the user assumes from the context. It is largely a UI design challenge to create a UI where the mismatch does not lead to problems.

Multitouch evolutions


One interesting comment from Stèphane was that he and the Intuilab were building more multi-touch UIs a few years ago than he is now. Back then, the interfaces were for specialists who could afford training and thus complex interactions. These days, multitouch is mostly for ad-hoc interaction without any previous training on the UI. Thus, the UI must be "intuitive" and easy to understand. The result of that is that modern multitouch is more often than not just two-finger touch. This also matches the (little) experience I've had with the iPhone so far.

Another thing I noticed again on our tour through the Intuilab is that multitouch interfaces are mainly aimed for consuming content. Some interfaces to assemble existing content are used (photos, slides, etc.) but real content creation seems rare and still relies heavily on settings with more traditional input devices.

A tour through the ENAC LII showed another interesting change - a system for ATC training that was using touchscreens exclusively a few years ago has since been switched to use Wacom Cintiq tablets with styli. As with any device, multitouch devices are no silver bullet, they are great for some tasks but not a complete replacement for all other devices. Using them appropriately is the key to success. Bill Buxton's often-cited mantra of "Everything is best for something and worst for something else" applies here as well.

Apple, Microsoft and other multitouch products


Why is it taking us so long when there's plenty of multitouch offerings out there already? The simple answer is: we are not working on the same problem.

If we look at commercial products that provide multitouch, Apple's iPhones and iPads are often the first ones that come to mind. These provide multitouch but in a very restrictive setting: one multi-touch aware application running in full-screen. Doing this is suprisingly easy from a technical point of view, all you need is a new API that you write all new applications against. It is of course still hard to make it a good API and design good user interfaces for the new applications, but that is not a purely technical problem anymore. Apple's products also provide multitouch in a new setting, an evironment that's closer to an appliance than a traditional desktop. They have a defined set of features, different form factors, and many of the user expectations we have on the traditional desktop do not exist. For example, hardly anyone expects Word or OpenOffice to run as-is on an iPhone.

The main problems we face with integrating multitouch support into the X server is the need for the traditional desktop. Multitouch must work across multiple windowed application windows, with some pointer emulation to be able to use legacy applications on a screen. I have yet to see a commercial solution that provides this, even the Microsoft Surface applications I've played with so far only emulate this within very restrictive settings. (Disclaimer: I do not own an iPhone, iPad or Surface table so there is the chance that I've missed some new development). We cannot simply ask all developers to rewrite their applications as multi-touch aware and completely ditch the existing ones. Note that e.g. Android can do this, for the same reasons that Apple could.

Having multitouch work across any application window, with fluid transitions between pointer/keyboard interfaces and multitouch interfaces is tricky. And of course, we need to stay compatible with the core protocol and its extensions. Funnily enough, many of the issues we face with multitouch are similar to the ones I faced years ago during the MPX development phase. This is one reason why I am more confident but also quite cautious about it, I already know how it can go wrong.

The basic requirements we have are:

  • Multitouch in any application that supports it

  • Multitouch in any number of applications simultanously

  • Pointer emulation for non-multitouch applications

  • Simultaneous execution of multitouch and pointer/keyboard applications



Replace multitouch with multiple users and you have exactly the same requirements I had for MPX (and XI2). Why exactly these four requirements I'll explain separately one day, but the short answer is: zero transition cost between multitouch and non-multitouch interfaces.

Gestures


Whenever multitouch is mentioned, gestures are brought up quickly too. Gestures are prevalent in this context because they appear to be a natural extension of touch input. They are, but only to a very limited extent. The set of natural gestures is surprisingly small and goes barely beyond the well-known tap, pinch, rotate and swipe. As an exercise, think of the natural gestures for copy and paste. Then see how many of your friends come up with exactly the same gesture.
Moreover, gesture discussions often focus on multi-finger gestures. In the not-too-distant future off-the-shelf hardware will support arbitrary object shapes and that's when hand and arm gestures will become relevant too.

Gestures do not rule out touch. Especially on direct-touch devices it is likely that a UI will use gestures as shortcuts to complex operations but the vast majority of interactions just use plain touch input for object selection, moving objects, and invoking menu commands. So it is important to think of gestures as an abstraction of some touch input but it is by no means the sole vehicle of deliverying touch input. From a technical point of view, this means that gesture recognition must not be in the way of touch recognition.

Gesture detection introduces a delay and that is unavoidable. What can be avoided is the effect of this delay on the UI. With good gesture design the delay is irrelevant. Take the example of a system with two gestures: a one finger hold gesture and a four finger hold gesture (i.e. an action is triggered when one or four fingers are put onto a position). Assume that the timeout for the gesture is 2 seconds. When a user puts a single finger on the device, the system now has to wait 2 seconds before it can react. If within 2 seconds the user does not put three more fingers down, the single-finger action can be taken. Decreasing the timeout lands the UI designer at the blurry line between a laggy UI and too-hard-to-trigger gestures.

To solve this issue, the system must be able to recognize which gesture is performed before the user expects feedback. Consider the gesture set of tapping one finger, moving one finger and a four finger hold gesture. Tapping and moving are easy to detect and they don't suffer from an artificial delay. The only time the timeout is needed is when the user puts a finger down but doesn't move or release it immediately. Now the system has to wait for more fingers. However, since a single finger not moving is not a defined gesture, the user expects no feedback and the timeout is irrelevant.

Integrating gesture support is thus not a matter of simply defining a few gestures. It's a matter of selecting the right ones that do not expose the user to a laggy experience. The FOSS software stack is notoriously segregated and thus true gesture integration must be done across window managers, desktop environments and applications. Coincidentally, this is my main worry with the Unity gesture support, I feel the gestures have a potential to overlap in some cases and thus be detrimental to the user experience. I hope I'm wrong here.

X.Org multitouch support


When it comes to input event hanlding, the X server's job is to convert a set of hardware events into a set of events meaningful in a GUI. This includes converting the hardware information into screen-relative information and picking (selecting where to send the event to). I've already outlined the four requirements for multitouch above.

Originally, I thought multitouch could be integrated into our current event system but a few months back I gave up on that. Whenever I tried to sort out the corner cases, there were some situations where the core protocol would be unresolvable. The current aim is that we add new events and they will live side-by-side to pointer and keyboard events, not necessarily influencing those. This means that clients must add support for multitouch to be useable, even XI2 enabled clients will not provide that functionality without modifications (pointer emulation for single-finger touches is still considered as required feature though).

Beyond that, things are still in flux. At the time of this writing, there are two protocol proposals that are largely overlapping. Daniel Stone's version has taken a few features from mine as I have taken some of his but the basics approach is the same (and we came up with it independently, which is a good sign). Daniel's proposal is also backed by an implementation though I have yet to look at these patches. Either way, what will be happening is a new set of events and a few requests to detect which devices have multitouch. Passive grabs on touch points will be possible but that still needs to be sorted out. Once it is, I'll post more details.

The summary here is that things are moving, at least in my case largely due to Canonical's announcement which lit a fire under some southern body parts and finally made me spend more time thinking about multitouch.

Ubuntu's multitouch support


One thing that is likely of interest to many is how all this will integrate with Ubuntu's multitouch support. Ubuntu has committed to strong gesture support. This is commendable but quite limiting in my opinion as true multitouch is much more than gestures. The current UI seems aimed more at dependent-touch devices like the Apple Magic Trackpad where gestures agument traditional input, less so at a fully multitouch-aware desktop. Fair enough, so far there aren't a lot of multitouch applications out there but for system-level design we need to keep the medium and long term in mind too.

The current integration is a hack (and the developers don't try to hide this), but most good things start as hacks so there's not much to say here. I am certianly impressed that they are willing to deliver a UI based on what is still a moving target. The few issues I had were with the design, so let's focus on that.

The originally proposed X Gesture Extension tried to get a recogniser into the server (not a 100% accurate statement, but close enough). It only allowed for one recogniser in the system. I do not think that we'll ever manage to find the one true gesture recognizer, we haven't managed to find the one true window manager or toolkit either. I talked at length to Chase about this and the revised architecture diagrams Chase presented at the workshop and XDS do not call for gesture recognition in the server anymore.

With the new approach the recognition happens purely client-side and is done by a library or a daemon. This allows for multiple different gesture recognisers be active at the same time, a scenario that is quite likely to happen (I do envision GTK, Qt, Mozilla, etc. all wanting their own system). Whether that's a good thing for the UI is another matter, consistency for gestures is important and especially Ping Cheng was not happy at the prospect of having multiple, possibly inconsistent, systems. We need to find some common ground here between desktop environments and toolkits. Geis tries to address this, though it is in a quite early stage. The main issue with geis is a political one. The implementation requires copyright assignment and puts up quite a barrier to entry. The utouch team tries to get around this by separating the interface specification and the implementation, but that's just calling for a fork or duplicated efforts before there's anything to fork or duplicate. So while I think geis is the right approach and much needed, I fear the political decisions will hold it back unnecessarily. My suggestion is to ditch the copyright assignment, because something as central as a generic gesture interface should be developed together with toolkit developers, X.Org developers, application developers, etc. And that's just easier when contributions are as simple as possible.

As for the UI itself - any UI decisions already made will unlikely be an issue from a technical perspective. Gestures are an important part of multitouch support and we will make it possible to provide gestures. All the technical parts are still in flux, but the current abstractions Ubuntu has put in place are likely good enough to adjust to a new baseline. So while I cannot speak for Ubuntu at all, my guess is that in future versions the bottom bits of the stack will be swapped out but the top bits (the user-visible ones) will largely remain the same.

Conclusion


The above should provide a general overview on some of the issues with multitouch and challenges we face. I've skipped over a few and only briefly touched on others for brevity, it's quite easy to write full novels about all this.

I will post more technical details about the kernel's multitouch API and how we plan to use it in X.Org for our protocol extensions once the latter is at least partially nailed down. For now, the concepts described above hopefully framed your mind in terms of multitouch and some of the concepts we're looking at.

Wednesday, September 8, 2010

Presentation in Toulouse

I'll be giving a talk next Tuesday, 14 September in Toulouse, France. If you happen to be around that area, more information is available here:

http://www.toulibre.org/
http://www.linuxfr.org/2010/09/07/27352.html

Friday, September 3, 2010

Wacom support in Linux

This article should have been written a long long time ago. Anyway, here's the gist of Wacom tablet support under Linux and that hopefully clears up the confusion about the various packages.


The role of linuxwacom


linuxwacom is two things. It is the project name used for wacom-specific implementations and thus in the URL as well (in this post, I will always use "linuxwacom project" to refer to the project).

It is also the name of the tarball shipped by the linuxwacom project. Or at least the name of one tarball, anyway. Now, this tarball includes virtually everything linuxwacom as a project wants to provide, most notably a kernel driver, an X driver and several utilities. More on these later.

The main (and long-term) contributor to linuxwacom is still Ping Cheng, a Wacom employee. Wacom naturally has a strong focus on its customers and thus linuxwacom goes into some pains supporting systems all the way back to XFree86-4.

However, the X server has moved quite a bit.

xf86-input-wacom


In August 2009, I split out the X11 driver from the linuxwacom package and imported it into a git repository called xf86-input-wacom. This driver has seen quite a bit of rework, including dropping of some features (like XFree86-4 support but also old-style serial tablets) and gaining some features (like input device properties). The driver includes a simple tool called xsetwacom that provides some options to manipulate driver settings at runtime. xsetwacom used to be part of the linuxwacom tarball and I've tried to keep xsetwacom's interface as close to the original as possible.

As of a couple of weeks ago, the xf86-input-wacom driver also includes a simple isdv4-serial-debugger to print information about serial tablets. That's it though, xf86-input-wacom does not contain any other tools.

xf86-input-wacom supports serial devices directly and USB devices provided the kernel supports them. In that regard, it is quite similar to evdev or synaptics - both rely on the kernel for actual hardware support. If your kernel doesn't support a particular device, then xf86-input-wacom won't help.

Note that xf86-input-wacom is the replacement for the linuxwacom X driver, that driver will not compile on X servers 1.7 or later. And the name does not imply that it is for Wacom tablets only, we have code to handle some Waltop tablets and recently added a patch for Fujitsu serial devices. More devices will be added as needed.

kernel drivers


The linuxwacom tarball ships with several kernel drivers for various kernel versions. Note that these are out-of-tree drivers and thus if you compile them and then update your kernel, you must recompile and re-install the driver. This out-of-tree development is mostly due to Wacom's commitment to its customers.

Personally I would love to see kernel development happening on LKML only but this is not the case right now. Especially for newer models that support touch the upstream kernel requires the use of the MT protocol, something that the X drivers (neither xf86-input-wacom nor the old linuxwacom one) can handle right now. Especially Bamboos are affected by this. AFAIK, no distribution supports them out-of-the box and users must compile the kernel driver from the linuxwacom tarball. Once that driver is installed, xf86-input-wacom works fine with it.

This is a source of much confusion but while I intended to start working at the kernel drivers, I have enough things on my slate and I have yet to actually do so. Luckily, Henrik and others are working on it.

However, when I read "You may be tempted to download the 0.10.8 version if you have an X server >= 1.7. I tried this, but it did not work. Stick with the production driver, it will work." and (after compiling the kernel driver) "Now, plug in your tablet. It SHOULD be recognized and immediately function as a mouse. " on linux.com I cringed. What happens here is that after the kernel driver from the linuxwacom tarball was installed, xf86-input-wacom automatically picked it up. You need both parts, kernel and X driver.

wacdump, wacomcpl, and other tools


All these are still available in the linuxwacom tarball. wacdump provides similar functionality to evtest so I don't see the need for maintaining two tools that do the same thing. xidump does the same as xinput --test. Hence both tools are not available anymore on newer distributions.

wacomcpl is a major source of headache. It's a TCL wrapper around xsetwacom to provide a basic graphical configuration utility. It doesn't work anymore due to a number of subtle changes in the xsetwacom interface but also because it partially expects options that don't exist anymore (or in other forms) in the xf86-input-wacom driver. I thought about fixing it up but after looking at it I ran away screaming. I've traded some bugs with Bastien and he'll write a GNOME configuration utility. That's currently holding up on the need for an actual UI (and as engineers we both shouldn't do that one ourselves...) but it will hopefully see the light at some point in the near future. Meanwhile, no GUI configuration tool.


Future progress


The linuxwacom project is still largely controlled by Wacom, with the exception of xf86-input-wacom that's maintained by me but is heavily contributed to by Ping Cheng, both in patches and code review.

Wacom's internal processes are different to the ones we use in distributions and upstream and that can make life hard for us. (Likewise, our processes make life harder for Wacom so the finger-pointing goes both ways :) Ping and I have had plenty of discussions about this and we try to meet somewhere in the middle where possible, disrupting the project completely helps no-one.

xf86-input-wacom is in a reasonably good state now though the cleanup has taken much longer than expected and there are still several items to be removed or rewritten to fit into the new X server behaviours. I expect this work to take another couple of months but much of it is aimed ad making the driver easier to maintain and develop for. For the average user, it won't have as much impact.

As I said above, I'd like to see kernel development happen on LKML in upstream kernels, not in a separate tarball. This is - aside from the GUI tool - the most pressing issue right now.

How to help


As usual, the best way of helping out is to test the code that is available. I'm not a designer so if the tablet doesn't crash the server that's pretty close to 100% functional to me. I rely on others to tell me when behaviours change.

If you're a coder, get right in. If your device doesn't work, go for the upstream kernel and try to get it working there. Anything upstream will feed back into distros and thus improve the out-of-the-box experience. If you have feature-requests for xf86-input-wacom, dig in and write it. The developer's mailing list is useful in any case. Finally, there's the wiki that will eventually be a useful one, but I keep getting distracted with other stuff whenever I try to move info over. So any help there will be useful to others looking for documentation.

And as I pointed out above - we're in desparate need for a UI for the GNOME config utility. If you're interested in helping out here, please contact me.

Finally, I want to point out that Ping (and by inference Wacom Inc.) has been excellent and though we sometimes disagree on technical issues there is no question that Wacom is supportive of the project and their contributions are invaluable.

XTS census

Tiago's X 1.9 Census includes the commits to the X Test Suite for the 1.9 window. IMO, this isn't particularly expressive since XTS follows its own cycle and the server merge windows have no meaning to it.

The short history of XTS is:
The X Test Suite was written back in the day of yonder (possibly more than two yonks ago) and it is designed to perform a number of protocol tests to verify whether the server has changed behaviour. This includes simple things like creating a window and making sure the MapNotify is sent through to more complex interactions with modifiers and grabs.

XTS was still in CVS until 2009 when I wanted to figure out how much I really broke with MPX. So I imported it into git and started autotooling it. XTS is a bit of a beast and at some point I gave up and asked Dan Nicholson for help; his autotooling skills exceed mine by some unmeasurable amount. A few months later and some 80-something commits later Dan came back with a fully autotooled XTS. So building XTS went from requiring a wiki page to git clone, autogen.sh, make.

From then on, we continued working on making it more sane and easier to run, along with misc code cleanups (DECNET support? Really?) and one of my favourite issues: that after 4 hours of runtime one realised that every test had failed because DISPLAY wasn't set. Anyway, I digress.
We found a couple of issues in the X server as well, with 1.7.7, 1.8.1 and 1.9.0 being the first three releases that pass the test suite without crashing.

XTS is still a bit from being really useful, at the moment many of the test results are write-only (though a few test errors have fed back into the server or even libxcb). Anyway, here are the raw numbers from gitdm (that 1 employer is Unknown because I couldn't be bothered setting up a gitdm.config file).

Note: 1024 of my changesets are a semi-automatic rename of files and that skews the statistic.


Processed 1275 csets from 5 developers
1 employers found
A total of 33380 lines added, 30758 removed (delta 2622)

Developers with the most changesets
Peter Hutterer 1061 (83.2%)
Dan Nicholson 192 (15.1%)
Aaron Plattner 12 (0.9%)
Jon TURNEY 4 (0.3%)
Tinderbox user 4 (0.3%)

Developers with the most changed lines
Dan Nicholson 20545 (50.0%)
Peter Hutterer 10296 (25.0%)
Aaron Plattner 3854 (9.4%)
Tinderbox user 655 (1.6%)
Jon TURNEY 30 (0.1%)

Developers with the most lines removed
Aaron Plattner 2000 (6.5%)

Developers with the most signoffs (total 5)
Dan Nicholson 3 (60.0%)
Peter Hutterer 2 (40.0%)

Developers with the most reviews (total 10)
Dan Nicholson 7 (70.0%)
Peter Hutterer 3 (30.0%)

Developers with the most test credits (total 1)
Pat Kane 1 (100.0%)

Developers who gave the most tested-by credits (total 1)
Dan Nicholson 1 (100.0%)


The point of this blog post? Give credit to Dan for his magnificent work on XTS because he's been carrying the load for quite a while. And of course, many thanks to Aaron, Jon and Pat for their contributions.

Tuesday, August 31, 2010

The antipattern of hate

Code changed to protect the guilty, but the approach is the same. Also, this pseudo-code below is actually more streamlined, more sensible and slightly saner than the original.


static char* array[10];
static int other_array[10];

void get_values(int n)
{
for (i = 0; i < n; i++)
other_array[i] = get_val(array[i]);
}

int main (void) {
array[0] = "somestring";
array[1] = "some other string"
array[2] = "lalala"
array[3] = "bananas"

get_values(4);

for (i = 0; i < 4; i++)
blah(other_array[i]);

return 0;
}


If you think of writing code like that, don't. Google for function parameters and try to understand how they work. Some poor sod will eventually have to figure out wtf you're trying to do [1]. I've encountered this bit of code a few weeks back and I am still flabbergasted.

/me hopes this venting makes the anger go away.


[1] as I stated in an early post, any software project is a collaborative project. It has at least two developers [...]. The person you may end up hating for the above may be you.

Friday, August 6, 2010

First draft of multitouch protocol spec published

In case you're not subscribed to xorg-devel, I've sent off the first draft for the multitouch X protocol specification (as part of the X Input Extension 2.1) today. If you have interest in multitouch, please read through it and see if you can poke holes into the current approach.

http://lists.freedesktop.org/archives/xorg-devel/2010-August/011759.html

Note that this is very low-level stuff. The data described by the protocol is what's sent on the wire to the client. Most applications are separated from this data by toolkits (GTK, Qt, etc.) and only few people are unlucky enough to have to care about the protocol itself.

Friday, July 2, 2010

Input event processing in X

The terms Core Pointer, VCP, extension devices, etc. are thrown around much these days. But they either mean very little or just very little to most. Especially because since X server 1.7 and to a lesser part 1.6, the definitions have changed a lot. So here's a bit of a high-level overview of what happens with input events in the server.

X input drivers


The first interesting point is probably that drivers these days aren't really drivers any more. The most common drivers, evdev and synaptics don't have hardware-specifics anymore, they rely on the (Linux) kernel to provide generic events. Thus, if a device isn't handled by evdev that's hardly ever an evdev bug - it usually means that the device isn't handled by the kernel (yet). Same goes for synaptics, if a new touchpad doesn't work it usually isn't supported by the kernel. The situation is different on Solaris and the BSDs, where the mouse/kbd drivers as well as synaptics have hardware-specific protocol parsers.

But on Linux, X input drivers are more sets of features than actual hardware drivers. evdev will happily manage your touchpad but you'll miss out on two-finger scrolling and other synaptics features.

X input devices


Any physical input device that's exported by the kernel as a /dev/input/eventX device file is usually added to the X server as an input device. For example on my box here I have the following devices:

:: whot@quokka:~> xinput --list --short
⎡ Virtual core pointer id=2 [master pointer (3)]
⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]
⎜ ↳ Microsoft Microsoft® Digital Media Keyboard id=8 [slave pointer (2)]
⎜ ↳ TPPS/2 IBM TrackPoint id=13 [slave pointer (2)]
⎜ ↳ Macintosh mouse button emulation id=14 [slave pointer (2)]
⎜ ↳ SynPS/2 Synaptics TouchPad id=15 [slave pointer (2)]
⎜ ↳ Microsoft Microsoft 5-Button Mouse with IntelliEye(TM) id=16 [slave pointer (2)]
⎜ ↳ Wacom Intuos4 6x9 eraser id=17 [slave pointer (2)]
⎜ ↳ Wacom Intuos4 6x9 cursor id=18 [slave pointer (2)]
⎜ ↳ Wacom Intuos4 6x9 pad id=19 [slave pointer (2)]
⎜ ↳ Wacom Intuos4 6x9 id=20 [slave pointer (2)]
⎣ Virtual core keyboard id=3 [master keyboard (2)]
↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)]
↳ ThinkPad Extra Buttons id=6 [slave keyboard (3)]
↳ AT Translated Set 2 keyboard id=7 [slave keyboard (3)]
↳ Microsoft Microsoft® Digital Media Keyboard id=9 [slave keyboard (3)]
↳ Sleep Button id=10 [slave keyboard (3)]
↳ Video Bus id=11 [slave keyboard (3)]
↳ Power Button id=12 [slave keyboard (3)]


At the bottom of the list we have devices like the Power Button and the Video Bus - they look like keyboards to us so they get added (one can usually match up /proc/bus/input/devices with the X device list). Note that it is the configuration deciding which devices get added, not the server itself. The default configurations we ship in the distros simply add all /dev/input/eventX files, hence the exhaustive list. It is possible to blacklist devices as well.

The actual input devices in the list above are obvious as well, the touchpad, the trackpoint, a wacom tablet and a mouse and two keyboards. From the X server's point of view, the touchpad is identical to a mouse btw., the fact that other drivers are used doesn't matter to the server.
This is my everday laptop and I have 18 devices, that's quite a bit. Our current maximum is 40 but that's just an arbitrary define that can be increased to 128. Once that level is reached, things get more complicated but I'll skip on the details here. For now, we're safe.

ID numbering starts at 2 for internal reasons, and in our implementation devices 2, 3, 4 and 5 are hardcoded and always present. More on that later. 0 and 1 are reserved for XI2, if you've programmed with it you may remember the XIAllDevices and XIAllMasterDevices defines. They are treated like devices internally.

What you can also see in this list is that the server differs between pointers and keyboards. This is an unfortunate but required distinction and we have code all over the place to handle pointer events from keyboards and the other way round. There are also plans to work around this distinction in evdev, but that requires some API work in the server.


Slave devices


What really matters in the server is not the physical devices but the so-called slave devices. This is a term that got introduced by the MPX work pre server 1.7.

  • A physical device usually corresponds to one or more slave devices. There are two exceptions here: the wacom X11 driver creates multiple X devices from a single device file. The other exception are physical devices that are split into multiple kernel devices. For example, all touch devices that need the kernel's multi-device quirk show up as several /dev/input/eventX devices and thus are added as multiple X devices. Nonetheless, for most devices, the physical device == slave devices is valid.

  • A slave device may not be a physical device at all (e.g. "Virtual core XTEST pointer", "Virtual core XTEST keyboard").

  • The server may at any point in time have zero or more slave devices. Our current implementation has two hardcoded ones (the XTEST pointer and XTEST keyboard) but they are not required by the protocol.

  • In the default configuration, all slave devices are hotplugged. Even when they're already present at server startup, there's no difference in the code path.

  • A slave device may be either attached or floating. More on that later.

  • Any slave device, once enabled, may generate events, but only X Input Extension events, both for versions 1.x and XI2. More on that later.



The last point is interesting for clients - if your client uses the core input API (e.g. XSelectInput(3)), then slave devices are invisible to the client. Currently, virtually all clients use core input, a situation that hopefully changes in the future. GTK for example will support XI2 in its next version.

Master devices


For indirect input devices like a mouse, the virtual representation of the device is as important as the device. We tend to think of "moving the cursor" as equivalent as "moving the mouse", those to concepts are quite ingrained. (Note that in this article, the term "cursor" applies to the concept, not to the shape). The same goes for the keyboard focus, but both cursor and focus are just abstract virtual representations. We represent them as devices as well, they are so-called master devices.

In the device list above, you can see two master devices - one pointer, one keyboard. Other facts about master devices are:


  • Master devices always come as pairs, one pointer and one keyboard. Internally, the devices know the respective other device in the pair.

  • Our implementation has two master devices hardcoded, the "virtual core pointer" and the "virtual core keyboard". More on that later.

  • Master devices do not generate events by themselves, they rely on slave devices to generate events for them (more on that later).

  • A master device may have zero or more slave devices attached.

  • You may have more than one pair of master devices - this is the essence of MPX.

  • Master devices may be created or destroyed by a client at runtime. Likewise, the attachment of slave devices can be manipulated at runtime.

  • Master devices may send core events and/or X Input Extension events (1.x and XI2). Only the first pair of master devices is visible to X Input 1.x clients. This is a protocol detail that I won't elaborate on here.



Now the terminology of attached and floating on slave devices may be a bit clearer, a floating slave is simply not attached to a master device. That affects event delivery, more on that later.

Again, the last point is of interest to the clients again - core events always and only come from a master device. These are the devices a client grabs when it grabs the pointer or the keyboard. These are the devices whose state is returned when the client asks for the modifier key state. And because core clients assume only one pointer and only one keyboard, the server has some elaborate tricks to make a client believe there's only one pair, even when we have multiple master devices present.

Event generation


Event generation is the first half of the event processing. When the kernel device file has new information for us, the server receives a SIGIO and calls into the driver. The input driver reads this information off, processes it and "posts" it up as key, button or motion data. The server then converts the data into a internal event format and puts it onto the event queue. Once that's done, the signal handler returns and processing continues as usual.

Depending on the driver, zero or more events may be posted for each kernel event. For example, the evdev mouse wheel emulation converts motion data into several button press and release events. Same with the synaptics driver that converts position data into button events for edge scrolling.

Event delivery


During event delivery, the events are taken out of the event queue and processed one-by-one in the order they were received. The event delivery itself is staged into slave delivery and master delivery.

As said above, an event may only be generated by a slave device. Once that happens, the event is processed for this slave device and delivered to any listening clients - if there are any. Remember, these clients would have to be XI 1.x or XI2 aware to get events. Regardless of the event delivery, the same event is then processed again but this time for the respective master device. Obviously, if the slave device is floating this stage is skipped. Looking at the device list above, whenever I press a button on the "SynPS/2 Synaptics TouchPad", this event is also delivered through the "Virtual core pointer".

When the event passes through the master device, it updates the master's internal state and is then delivered to the client. Delivery happens either as core event, as XI 1.x event or as XI2 event. The same event may be delivered to multiple clients (for example if two or more clients registered the same event mask on the same window), but once delivered the event cannot be delivered as a different event category.

Master devices are the union of the respective slave devices. Thus, if you have two mice attached to the same master device and press the first button on both, the button won't be released on the master until all mice have released that button. Likewise, if you press the left button on mouse A and the right button on mouse B, the master device appears to have both buttons down.

What about the CorePointer/CoreKeyboard?


Back before server 1.4, the CorePointer option was a common configuration option. The XI 1.x protocol specification requires that one pointer and one device be deemed the "core" devices, thus only able to generate core events and not XI 1.x events. All other devices may generate XI 1.x events but not core events. Configuring a device as CorePointer simply said: this device controls the cursor.
Daniel Stone's hotplugging work in 1.4 introduced the "Virtual core pointer" and the "Virtual core keyboard". Instead of a physical device being a core pointer, these hardcoded virtual devices became the default core pointer and keyboard, allowing all physical devices to be used as XI extension devices. Note that only XI extension devices provide for more than two axes, so e.g. pressure is impossible with the core protocol. All MPX did was to allow for more than one pair of these hardcoded virtual devices (surprisingly, that's not quite that trivial).

In the old system, there was the option of "SendCoreEvents". This option worked around extension devices not being allowed to generate core events. For any device that "SendCoreEvents", the core pointer would generate a core event whenever the original device generated an event. You may notice that this sounds quite similar to the staged event delivery and in fact it is (the implementation today is very different though). These days the SendCoreEvent option is only useful if you want to turn it off. And if that's the case, the device is simply set floating at server startup - because as said above, floating devices do not pass their events to the master device and thus do not send core events.

Finally, before 1.7 drivers could manipulate the "SendCoreEvents" setting in the device. This is not possible anymore, whether a device sends core events is only decided based on whether it is attached or not.

XTEST devices


The last two devices that may need explanation are the XTEST devices. They are hardcoded for each master device and serve as input devices for the XTEST extension. This extension allows clients to trigger pointer or keyboard event just as if a physical device has generated them. Whenever a client calls XTestFakeButtonEvent(3), it's the XTEST pointer that presses a virtual button down, generating an event on the master device.

Handling multiple master devices


I've explained this in the past but I want to touch on this again to have everything in one article to link to. The ClientPointer is a new principle introduced with MPX.

Many protocol requests ask for information or action on the "pointer" or on the "keyboard". With XI 1.x, this meant the core pointer and the core keyboard, respectively. With XI2/MPX in the server, there may be more than one core pointer or keyboard at any time. So when a client asks "where is the cursor", we need to reply with one of them. The ClientPointer is simply a reference to a master pointer device that each client holds. Whenever device information is requested but no device is specified, the server will take the ClientPointer (or the paired master keyboard). The general rule is - unless you reassign the ClientPointer (XSetClientPointer(3)), all applications will get information from the first pointer and keyboard device. There are a few tricks in the server though to dynamically reassign ClientPointers during grabs. This greatly improve usability on legacy desktops and is one of the reasons why you can - on a standard desktop - use two applications simultaneously with only minor glitches.


A simplified summary




  • Each physical device is added as a so-called "slave device" to the server.

  • Slave devices are attached to so-called "master devices".

  • A master pointer is equivalent to a cursor, a master keyboard is equivalent to a keyboard focus.

  • When a slave device generates an event, this event is sent through the master device as well.

  • A master device is the union of all attached slave devices.

  • Legacy clients only receive events from master devices.

  • There is no concept of a "core pointer" or "core keyboard" anymore, the master devices have taken over this role.

Wednesday, June 30, 2010

XTS has a new home

Gather round children, for I shall bore you with a story. Back in the olden days, we had a X Test Suite (XTS). Amongst other things, it tried to verify certain behaviours in light of protocol requests. It's quite huge, covering pretty much all of the core protocol and a few extensions as well. That beast, and a beast it was, was maintained in CVS and mostly dead.

In March 2009 we started autotooling it, removing cruft, general weirdness and over time it became more useful. Many thanks here to Dan Nicholson, Aaron Plattner and Jon Turney. Now we've finally moved it from various $HOME directories to a git repo proper. The work is not completed yet though. The code does show its age and many a cleanup work is needed to drag XTS into the future. Kicking and screaming may be involved, swearing will definitely be part of that process.

To get the test suite, run

git clone git://anongit.freedesktop.org/git/xorg/test/xts.git


It is now at a stage where it pretty much runs with the usual autoreconf/configure/make steps and spits out results. The usefulness of those results is in debate, they all need auditing. You must run it against a plain X server, running a window manager will alter the results.

Note that XTS is a developer tool, be aware that it's inner workings and output may be confusing. If you look at it long enough, you may see the matrix.

Tuesday, June 22, 2010

xmodmap keyboard deconfiguration

Reading this post on linux.com last weekend made me cringe. Not because it is factually wrong (it isn't), but because what's described here is the reason for a great many bugreports and some rather uninformed hatemail. One thing that startled me in particular is the hint that "to really take charge, you need to dig into the X keyboard extension" when the rest of the article doesn't deal with XKB at all. This prompted me to write this blurb here.

The abstraction cardhouse



In X, keyboard input is abstracted. The keyboard sends a keycode to the client. The client then translates that keycode into a keysym (a glyph, in most cases, more or less), given a keycode-keysym table. Keycodes should be treated as arbitrary by the client. Keysyms on the other hand are constant, the keysym for "♥" is always the same (0x1002665). In the core protocol, that lookup table was reasonably straightforward (for some value of "straightforward"), with several modifiers providing access to different levels in the table. The table being based on the max number of modifiers. For 105 keys and two modifiers, the table was essentially 210 symbols wide (modifiers can be "shift", "alt", etc.).

That was deemed to be insufficient, causing the development of the X Keyboard Extension, lovingly referred to as "XKB" or "that bloody nail in my coffin". I won't go into details, but the lookup tables are a tad more complicated. XKB is integrated around core so that XKB and core applications can coexist nicely. Up to a point that for a long time we kept XKB state and core state separately inside the server instead of generating it on-the-fly. What's unfortunate though is that the XKB protocol spec describes how to get the core lookup table from the XKB data. It also describes how to extrapolate from core to XKB. It doesn't however cover the full circle. So if you go from XKB to core back to XKB, guesswork happens.

Anyway, what you need to take from this is that the general rule of thumb is - any commandline tool uses core, any tool that allows you to select a layout, variant or model uses XKB.
And core doesn't support device specifiers, so it can only ever work on the default device or core device, more on that below.

Pulling out the cards



All the above was mostly good until 2007. That's when input hotplug happened. And 2009, when MPX happened. Now none of the old tools work properly anymore. Not because we've broken them, they still do what they've always done. But the environment has changed and what they do isn't sufficient anymore. So before you think about writing an angry comment that we've broken your configuration: stop using input and output hotplugging, stop using suspend/resume, stop using a desktop environment that manages your keyboard configuration. Then you'll notice everything still works as before and you can xmodmap away.

So why did everything break since?
Back before input hotplug we had one device that was deemed the "core keyboard". Whenever anything configured the device it set the configuration on this core device. Since xset, xmodmap and friends work on core only, the XKB configuration was then modified from that core table. All that was fine, because the core keyboard was hardly ever changed. Now with input-hotplug and MPX, we have the physical devices, attached to what is now called "master devices" but still serve the same purpose as the old "core keyboard". However, we've greatly increased the flexibility of the devices and physical devices are more often than not configured directly now. What we then end up with is a master device that has multiple keyboards configured, all with possible different keyboard configurations. In fact, non-US layout users in Fedora 12 are practically guaranteed to have multiple keyboard configurations. We only apply the user-selected layout to actual keyboard devices, not to devices with keys (Power Button, Video Bus, etc.). These devices come up with the "US" default.

In the case of a user with two physical keyboards, one qwerty, one azerty, the matching master device needs to adjust the keyboard configuration of the device in use. This means that it will switch between the two configurations as the devices are used. That works fine too. But if you now use xmodmap to change the the keyboard config on the master device, this configuration only stays as long as the same physical keyboard device is being used.
Now, the simple fix for this is to apply the new configuration to all devices instead of just the master device. And that's exactly what we do. However, this doesn't work with hotplugging. Once a new device is plugged in, it comes up with the configuration configured in the xorg.conf again. And that config doesn't include whatever xmodmap settings were applied later. So once that device is used, the settings are lost again or at least lost on this device.
Now sprinkle into this the fact that core doesn't really support multiple groups (layouts) on the same keyboard, that keycodes may be different between devices, that with MPX a device that was a slave to an xmodmap configured master device may end up being attached to a master that has a different XKB configuration and you've got the start of an interesting career.

The fix for all this is simple: make the session handle all keyboards, existing and hotplugged ones, and let it update the config accordingly. Whenever you're using a tool like xmodmap it is your responsibility to reapply the config after changing keyboards. Which brings me back to the article above - it curiously didn't mention that. So I strongly recommend that instead of configuring your keyboard with xmodmap you spend your time helping your favourite desktop environment to provide tools to cover your use-case.


PS: surprisingly "deconfiguration" doesn't exist in the dictionary but it seems the best word to describe what xmodmap does.

Wednesday, June 16, 2010

An (incomplete) roundup of touchpad features

Every once in a while, I stumble across a blog post titled "Multitouch with the synaptics driver" or something similar. Usually I find out that these posts describe how to enable two-finger scrolling or something similar. So I figured, maybe it's time to have something like a roundup of the features of the synaptics driver. As I have posted not too long ago, we have a great many options not all of them as useful as they could be.

First of all - the name "synaptics" is historical and now woefully inadequate. On Linux, we now rely on the kernel to address device-specifics and simply use the event API. In fact, synaptics is just like evdev with different features. (Note to non-Linux users: we still have the backends for hardware communication in the driver as well)

So whenever we talk about "synaptics devices", the "synaptics driver", or "synaptics features" in X, we mean "touchpad". All of the features are described in the man page and can be set either as an xorg.conf option or at runtime by tweaking the properties (with xinput or synclient). I'll brush on the capabilities exposed by the GNOME tool, for the KDE tool check the comments, I'm sure Kevin Kofler will point to them as usual.

Furthermore, I won't list the various options required to configure all this - simply look it up in the man page and you may find some other gems in there too.

Edge scrolling


The most commonly used touchpad feature is likely edge scrolling. Four edges are defined on the touchpad and movement outside of these edges is interpreted as scrolling. By convention, scrolling in X is buttons 4,5 and 6,7 for vertical and horizontal scrolling, respectively. If movement in the scrolling areas is detected, the driver converts the motion into a number of button presses for these scroll buttons.

Vertical and horizontal edge scrolling is hardcoded to only work on the right and the bottom part of the touchpad. They can be enabled separately and independently, though the GNOME GUI we have for it ties the horizontal scroll method to the vertical scroll method.

Edge scrolling is quite configurable, allowing for minimum and maximum speed settings and even pressure-dependent scrolling. Of course, the distance required to emit one scrolling event is configurable as well. None of these tweaks are exposed in the GNOME GUI.

Two-finger scrolling


Two-finger scrolling is the basic multi-touch feature that the driver provides. If two fingers are detected on the touchpad (you will need the hardware capabilities to do so), vertical and horizontal two-finger movements are converted into scrolling events. Provided the scroll methods are activated of course. The GNOME GUI allows for either edge or two-finger scrolling, the driver could provide both simultaneously.

Now, the interesting thing about two-finger scrolling is that it is usable on touchpads that only support single-fingers as well - through the two-finger emulation. If enabled, the driver tries to guess based on the width of the finger whether it is a single or dual-finger input and then trigger the required bits.

Other than that, two-finger scrolling is rather unexciting, it just does what it says on the box. As with edge scrolling, the minimum distances for a scroll event to be generated is configurable (but not in the GUI).

Multi-finger tapping


Tapping is the action of quickly putting a finger down on the touchpad and lifting it again. Multi-finger tapping is the same action with more than one finger. Synaptics currently supports up to three-finger tapping plus corner tapping. Each multi-finger tap can be assigned a different button, the default the GNOME tool assigns if tapping is enabled is 1/2/3 finger tapping to left/right/middle. Again, multi-finger support relies on your hardware.
Tap-and-drag is automatically enabled, whereby tapping an object, then dragging it with one finger "locks" the mouse button on the object.

Corner tapping works similar, but instead activates hot-zones in the corners of the touchpad (defined by the edge settings). Each corner can also be assigned a different button action. Corner tapping is not exposed in the GUI at this point.

Click fingers


Click fingers are similar to tapping in configuration and effect, but the trigger is different. The ClickFinger actions are executed when the left button is pressed while fingers are down on the touchpad. So if you leave two fingers resting on the touchpad and press the physical button, the configured action is executed. This can be quite useful for those that cannot or do not want to use tapping but still require left/right mouse button presses from their touchpad.

Circular scrolling


Similar to the edge and two-finger scrolling but detects circular motions instead of up/down and left/right motions. This works quite similar to the iPod interface. Depending on the trigger, a circular motion started in that trigger area (e.g. top right corner) will initialize the scrolling behaviour. Note that the circular scrolling events are generated per angle motion, i.e. by changing the radius of the circle you can change the speed of the scrolling.

Coasting


For scrolling, synaptics has coasting available as well. If enabled, scrolling continues even if the finger has lifted off the pad - provided enough force was to begin with. So if you quickly swipe two fingers up/down, scrolling will only stop once you tap the touchpad again. Coasting is disabled by default and judging by my computer, Firefox seems to be slow enough to give it a coasting feel anyway ;)
Also note that this isn't a physics model, it's a simple start/stop mechanism which arguably isn't quite as useful as a physics model may be.


Multi-touch gestures


We don't have any of the well-known pinch, rotate, swipe, etc. gestures yet. The main issue here is not the driver itself, it would be reasonably easy to add the bits to the driver but we can't do much with it. We have no meaningful way to transmit the gesture data to the client. So it's down to hacks like in the infamous elantech driver that shipped with the Dell Minis. Unfortunately, having the driver generate keystrokes like Ctrl+ to zoom in is hacky at best and nightmarish at worst.
We really need to update the middle-man here (the X server) to provide this information to the client so they can do the appropriate stuff with it. Alas, this work is taking a while.

SHM configuration


I just wanted to add this to have one more place to say it: SHM config doesn't exist anymore. You will need X server 1.6 or later to run the current versions of synaptics and if you see a message asking you to enable SHM configuration, your distribution may need updating or the program that pops up this message may need to be fixed.

Tuesday, June 15, 2010

driver configuration option madness


xf86-input-synaptics $> git grep "xf86Set.*Option" | wc -l
71


xf86SetIntOption, xf86SetBoolOption, etc. are the server's API to query configuration options. The touchpad driver has 71 one of them (rough count, one is counted twice but three are missing due to abstraction, so it's actually 73 then). I wonder how many of them are actually in use.

Carrying them around isn't really expensive. Good programming practice is to have all these set through variables or defines anyway so you might as well make those configurable. But still, over 70 options in a single driver?

At some point, we should start focusing on making the driver better by default instead of just throwing new configuration options at it. Any time the user has to change from the default configuration, we have failed.

Tuesday, June 1, 2010

New synaptics acceleration mechanism

In 2008, Simon Thum rewrote the X server pointer acceleration code into a complex mechanism that supported better tweaks for a variety of devices. However, a side-effect of this was that synaptics touchpads got accelerated twice, making pointer movement very fast and IMO quite imprecise.

A few months ago, Simon wrote a synaptics patch to pop a synaptics-specific acceleration mechanism into the driver to improve this situation. It's been delayed by me because although I loved it from the first minute onwards, I wanted to get an evaluation done to compare the before and after pointer behaviour in on some objective way (I did get as far as writing nearly all of the code required). Today I gave in and pushed the patch. This study isn't going to happen and I don't feel like holding back something that I believe is a real improvement to how touchpads feel in X.

Now, if you update to synaptics git the first thing you'll notice is that the pointer feels slower. After a few hours or a day you'll likely be used to it, so give yourself the time to get used to it. The new pointer accel code will be in the 1.3 release, which is due out soon (latest for X server 1.9). I'll try to find the time to add some knobs to the GNOME configuration tool by then too.

New evdev middle mouse button emulation defaults

As a heads-up, the next version of evdev (2.5) has a changed default for the middle mouse button emulation code. It is now off by default.

First of all - what is middle mouse button emulation?
Back in the olden days, when oil was cheap and mouse buttons were the peak of technology and thus expensive, mice only had two buttons. Possibly also to confuse user because two buttons for a (on average) five-finger hand seems just the right thing to do. Anyway, I digress.

One version of copy/paste in X relies on the middle mouse button, so a way to emulate that button was implemented in the driver. When this emulation was on, a left button was held back for a timeout and if during this timeout the right button was pressed, the driver would instead post a middle button event. If the right button didn't get pressed in time, the left button event was sent after the timeout expiry.

Coffee addicts could set this timeout to multiple minutes enough to have an excuse to fuel their addiction whenever they tried to do anything with the GUI. Now, two things have happened. The price of oil has surpassed the price of mouse buttons and on virtually all mice we have shiny things like scroll wheels, side buttons and whatnot. Also, computers have gotten a lot faster and people get annoyed about lag in the UI. Holding back a mouse button event by a few milliseconds seriously reduces the number of tweets one can write.

For the last couple of releases, we had the emulation on "auto" by default. It was on, until a physical middle mouse button click was detected. At this point the driver realised the emulation wasn't needed and turned it off. Which made for interesting side-effects: if you didn't know this feature, the GUI felt snappier after a middle mouse button press - for no obvious reason.
The "auto" option was a permanent false positive, there's no really reliable method to detect if a mouse has a middle mouse button.

So I've removed that "auto" feature and defaulted to off. Result - a slightly snappier feel of the UI for a quite underused feature. Of course, there are plenty of users of this feature, so it wasn't removed completely. If you need middle button emulation, please enable it with an xorg.conf snippet or ask your favourite desktop environment to provide a convenient checkbox.


Section "InputClass"
Identifier "middle button emulation class"
MatchIsPointer "on"
Option "Emulate3Buttons" "on"
EndSection

Monday, May 31, 2010

How to enable tapping (tap-to-click) on your touchpad in GNOME

Let's hope my Google-juice washes all those tap-to-click guides off the front page because - well, forget about most of them.
Here it is - the definitive guide on how to enable tapping on your touchpad in GNOME:

Go to System->Preferences->Mouse, select the Touchpad tab and then click "Enable mouse clicks with touchpad".

Or, because a picture says more than 17 words (emphasis mine):


The tap-to-click settings are one-finger for left click, two-finger for right-click and three-finger for middle-click. There's calls (and patches in review) for improvements to make this more configurable, but for now this is what we have.

And a quick run-down of what the other options mean:

  • "Disable touchpad while typing": spawns off syndaemon. syndaemon will monitor your keyboard and disable the touchpad while you're typing to avoid accidental input.

  • Scrolling options are: disabled, edge scrolling (where moving a finger on the right edge will scroll) and two-finger scrolling (two finger movement scrolls). In my case - because my touchpad doesn't support multiple fingers, the two-finger scrolling is greyed out.

  • "Enable horizontal scrolling": enables horizontal scrolling in the same scrolling method. For edge-scrolling this is the bottom edge, for two-finger scrolling it's anywhere on the touchpad with two fingers.



Things you don't need include editing the xorg.conf file, udev rules file, HAL fdi files, gsynaptics or other magic tools. You can still do that if you want to but you really have to want to.

[edit]
As pointed out in the comments, sometimes the control panel doesn't show a touchpad tab. The tab is conditional on whether a property initialized by the X.Org synaptics driver could be detected. So if the tab is missing, the usual cause is that the touchpad isn't detected correctly by the kernel and looks like a mouse to userspace. This can get confusing if the touchpad does tapping and other features in hardware but to userspace a tap looks like a left button press, etc. The latest versions of elantech touchpad suffered from this.

Wednesday, May 26, 2010

GTK merged XI2 support

From GNOME Bug 596725 - Add XInput2 support:
- Comment #11 from Matthias Clasen <...> 2010-05-26 00:07:29 +UTC ---
Merged


Nice!
Now I just need to find time to play with it...

Wednesday, March 3, 2010

Vodafone Australia mobile broadband and Fedora Linux

well, that title should get me a few hits :)

Anyway, the other day I had to set up one of these 3G mobile internet USB modems on a Fedora 12 box. The dongle in question was from Vodafone, purchased about a week and a half ago. It didn't work out of the box, though in hindsight it should have. So for anyone who's googling around on how to set this up (or whether it can be set up at all), here's the gist.

The dongle is a HUAWEI Technology usb modem, Vendor number 12d1, product id 1520. The modem looks like a mass storage device on account of the manufacturer trying to be smart. If you plug the modem in, dmesg has this to say:


usb 1-2: new high speed USB device using ehci_hcd and address 6
usb 1-2: New USB device found, idVendor=12d1, idProduct=1520
usb 1-2: New USB device strings: Mfr=3, Product=2, SerialNumber=0
usb 1-2: Product: HUAWEI Mobile
usb 1-2: Manufacturer: HUAWEI Technology
usb 1-2: configuration #1 chosen from 1 choice
scsi5 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 6
usb-storage: waiting for device to settle before scanning
usb-storage: device scan complete
scsi 5:0:0:0: CD-ROM Vodafone CD ROM (Huawei) 2.31 PQ: 0 ANSI: 2
sr1: scsi-1 drive
sr 5:0:0:0: Attached scsi CD-ROM sr1
sr 5:0:0:0: Attached scsi generic sg4 type 5


As I said above, these modems look like mass storage devices and need to be switched to the actual modem mode. The nifty program usb_modeswitch does exactly that, but alas, only in version 1.1.0 for this particular device. This version also includes the udev rules to make everything work automagically. At the time of writing, Fedora 12 didn't have 1.1.0 available yet (Bug 563503). So you can poke the maintainer to update the package (see the bug) or grep the spec file from that bug report and build the rpm yourself. Once built, install, plug the modem in and the dmesg looks a lot better:


usb 1-2: new high speed USB device using ehci_hcd and address 8
usb 1-2: New USB device found, idVendor=12d1, idProduct=1465
usb 1-2: New USB device strings: Mfr=3, Product=2, SerialNumber=0
usb 1-2: Product: HUAWEI Mobile
usb 1-2: Manufacturer: HUAWEI Technology
usb 1-2: configuration #1 chosen from 1 choice
scsi7 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 8
usb-storage: waiting for device to settle before scanning
scsi8 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 8
usb-storage: waiting for device to settle before scanning
usbcore: registered new interface driver usbserial
USB Serial support registered for generic
usbcore: registered new interface driver usbserial_generic
usbserial: USB Serial Driver core
USB Serial support registered for GSM modem (1-port)
usbcore: registered new interface driver option
option: v0.7.2:USB Driver for GSM modems
option 1-2:1.0: GSM modem (1-port) converter detected
usb 1-2: GSM modem (1-port) converter now attached to ttyUSB0
option 1-2:1.1: GSM modem (1-port) converter detected
usb 1-2: GSM modem (1-port) converter now attached to ttyUSB1
option 1-2:1.2: GSM modem (1-port) converter detected
usb 1-2: GSM modem (1-port) converter now attached to ttyUSB2
option 1-2:1.3: GSM modem (1-port) converter detected
usb 1-2: GSM modem (1-port) converter now attached to ttyUSB3
usb-storage: device scan complete
usb-storage: device scan complete
scsi 8:0:0:0: Direct-Access Vodafone Storage (Huawei) 2.31 PQ: 0 ANSI: 2
scsi 7:0:0:0: CD-ROM Vodafone CD ROM (Huawei) 2.31 PQ: 0 ANSI: 2
sd 8:0:0:0: Attached scsi generic sg4 type 0
sd 8:0:0:0: [sdd] Attached SCSI removable disk
sr1: scsi-1 drive
sr 7:0:0:0: Attached scsi CD-ROM sr1
sr 7:0:0:0: Attached scsi generic sg5 type 5


Whoopdi-doo. The modem is recognized as modem and we can go and set it up. Now, I don't get a warm fuzzy feeling when editing config files that I'll likely not look at again for a while, so I chose the clicky-pointy route: the magnificent NetworkManager. Click on the NM icon on your desktop, and it'll look like this:



Click on new connection, then jump through the wizard and you're done.










Now, all you have to do whenever you need to connect to the internet is plug the modem in, wait for a few seconds and then click onto your connection:




The sad thing is, that if usb_modeswitch 1.1.0 was installed already, it would have truly been plug and play, even though Vodafone claims that Linux is not supported. Oh well, at least this way I've got something to write about to shoo away those tumbleweeds on my blog.

Many thanks to Laurianne for letting my try the dongle on the Mac first (and letting me install the ridiculous piece of software Vodafone ships with the dongle) and Tom for running Debian which already was on usb_modeswitch 1.1.0 - showing that the modem works.