Thursday, March 27, 2014

Viewing the Xorg.log with journalctl

Those running Fedora Rawhide or GNOME 3.12 may have noticed that there is no Xorg.log file anymore. This is intentional, gdm now starts the X server so that it writes the log to the systemd journal. Update 29 Mar 2014: The X server itself has no capabilities for logging to the jornal yet, but no changes to the X server were needed anyway. gdm merely starts the server with a /dev/null logfile and redirects stdin/stderr to the journal.

Thus, to get the log file use journalctl, not vim, cat, less, notepad or whatever your $PAGER was before.

This leaves us with the following commands.

journalctl -e /usr/bin/Xorg 
Which would conveniently show something like this:
Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "wacom"
Mar 25 10:48:41 yabbi Xorg[5438]: (II) evdev: Lenovo Optical USB Mouse: Close
Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "evdev"
Mar 25 10:48:41 yabbi Xorg[5438]: (II) evdev: Integrated Camera: Close
Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "evdev"
Mar 25 10:48:41 yabbi Xorg[5438]: (II) evdev: Sleep Button: Close
Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "evdev"
Mar 25 10:48:41 yabbi Xorg[5438]: (II) evdev: Video Bus: Close
Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "evdev"
Mar 25 10:48:41 yabbi Xorg[5438]: (II) evdev: Power Button: Close
Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "evdev"
Mar 25 10:48:41 yabbi Xorg[5438]: (EE) Server terminated successfully (0). Closing log file.
The -e toggle jumps to the end and only shows 1000 lines, but that's usually enough. journalctl has a bunch more options described in the journalctl man page. Note the PID in square brackets though. You can easily limit the output to just that PID, which makes it ideal to attach to the log to a bug report.
journalctl /usr/bin/Xorg _PID=5438
Previously the server kept only a single backup log file around, so if you restarted twice after a crash, the log was gone. With the journal it's now easy to extract the log file from that crash five restarts ago. It's almost like the future is already here.

Tuesday, March 25, 2014

CyPS/2 Cypress Trackpad and firmware-based button emulation

For a longer story of this issue, please read Adam Williamson's post. The below is the gist of it, mostly for archival purposes.

The Dell XPS13 (not the current Haswell generation, the ones before) ships with a touchpad that identifies as "CyPS/2 Cypress Trackpad". This touchpad is, by the looks of it, a ClickPad and identifies itself as such by announcing the INPUT_PROP_BUTTONPAD evdev property. In the X.Org synaptics driver we enable a couple of features for those touchpads, the most visible of which is the software-emulated button areas. If you have your finger on the bottom left and click, it's a left click, on the bottom right it's a right click. The size and location of these areas are configurable in the driver but also trigger a couple of other behaviours, such as extra filters to avoid erroneous pointer movements.

The Cypress touchpad is different: it does the button emulation in firmware. A normal clickpad will give you a finger position and a BTN_LEFT event on click. The Cypress touchpads will simply send a BTN_LEFT or BTN_RIGHT event depending where the finger is located, but no finger position. Only once you move beyond some threshold will the touchpad send a finger position. This caused a number of issues when using the touchpad.

Fixing this is relatively simple: we merely need to tell the Cypress that it isn't a clickpad and hope that doesn't cause it some existential crisis. The proper way to do this is this kernel patch here by Hans de Goede. Until that is in your kernel, you can override it with a xorg.conf snippet:

Section "InputClass"+Section "InputClass"
        Identifier "Disable clickpad for CyPS/2 Cypress Trackpad"
        MatchProduct "CyPS/2 Cypress Trackpad"
        MatchDriver "synaptics"
        Option "ClickPad" "off"
EndSection
This snippet is safe to ship as a distribution-wide quirk.

Friday, March 21, 2014

Stacking xorg.conf.d snippets

We've had xorg.conf.d snippets for quite a while now (released with X Server 1.8 in April 2010). Many people use them as a single configuration that's spread across multiple files, but they also can be merged and rely on each other. The order they are applied is the lexical sort order of the directory, so I recommend always prefixing them with a number. But even within a single snippet, you can rely on the stacking. Let me give you an example:

$ cat /usr/share/X11/xorg.conf.d/10-evdev.conf
Section "InputClass"
    Identifier "evdev touchpad catchall"
    MatchIsTouchpad "on"
    MatchDevicePath "/dev/input/event*"
    Driver "evdev"
EndSection
$ cat /usr/share/X11/xorg.conf.d/50-synaptics.conf
Section "InputClass"
    Identifier "touchpad catchall"
    MatchIsTouchpad "on"
    MatchDevicePath "/dev/input/event*"
    Driver "synaptics"
EndSection
The first one applies the evdev driver to anything that looks like a touchpad. The second one, sorted later, overwrites this setting with the synaptics driver. Now, the second file also has a couple of other options:
Section "InputClass"
    Identifier "Default clickpad buttons"
    MatchDriver "synaptics"
    Option "SoftButtonAreas" "50% 0 82% 0 0 0 0 0"
EndSection
This option says "if the device is assigned the synaptics driver, merge the SoftButtonAreas option". And we have another one, from ages ago (which isn't actually needed anymore):
Section "InputClass"
    Identifier "Disable clickpad buttons on Apple touchpads"
    MatchProduct "Apple Wireless Trackpad"
    MatchDriver "synaptics"
    Option "ClickPad" "on"
EndSection
This adds on top of the other two, provided your device has the name and is assigned the synaptics driver.

The takeaway of this is that when you have your own xorg.conf.d snippet, there is almost never a need for you to write more than a 5-line snippet merging exactly that one or two options you want. Let the system take care of the rest.

Wednesday, March 19, 2014

X.Org synaptics support for the Lenovo T440, T540, X240, Helix, Yoga, X1 Carbon

This is a follow-up to my post from December Lenovo T440 touchpad button configuration. Except this time the support is real, or at least close to being finished. Since I am now seeing more and more hacks to get around all this I figured it's time for some info from the horse's mouth.

[update] I forgot to mention: synaptics 1.8 will have all these, the first snapshot is available here

Lenovo's newest series of laptops have a rather unusual touchpad. The trackstick does not have a set of physical buttons anymore. Instead, the top part of the touchpad serves as software-emulated buttons. In addition, the usual ClickPad-style software buttons are to be emulated on the bottom edge of the touchpad. An ASCII-art of that would look like this:

+----------------------------+
| LLLLLLLLLL MMMMM RRRRRRRRR |
|                            |
|                            |
|                            |
|                            |
|                            |
|                            |
| LLLLLLLL          RRRRRRRR |
+----------------------------+
Getting this to work required a fair bit of effort, patches to synaptics, the X server and the kernel and a fair bit of trial-and-error. Kudos for getting all this sorted goes to Hans the Goede, Benjamin Tissoires, Chandler Paul and Matthew Garrett. And in the process of fixing this we also fixed a bunch of other issues that have been plaguing clickpads for a while.

The first piece in the puzzle was to add a second software button area to the synaptics driver. Option "SecondarySoftButtonAreas" now allows a configuration in the same manner as the existing one (i.e. right and middle button). Any click in that software button area won't move the cursor, so the buttons will behave just like physical buttons. Of course, we expect that button area to work out of the box, so we now ship configuration files that detect the touchpad and apply that automatically. This requires an xserver fix and a kernel/udev fix, more on that later.

The second piece in the puzzle was to work around the touchpad firmware. The touchpads speak two protocols, RMI4 over SMBus and PS/2. Windows uses RMI4, Linux still uses PS/2. Apparently the firmware never got tested for PS/2 so the touchpad gives us bogus data for its axis ranges. A kernel fix for this is in the pipe.

Finally, the touchpad needed to be actually usable. So a bunch of patches that tweak the clickpad behaviours were merged in. If a finger is set down inside a software button area, finger movement does no longer affect the cursor. This stops the ever-so-slight but annoying movements when you execute a physical click on the touchpad. Also, there is a short timeout after a click to avoid cursor movement when the user just presses and releases the button. The timeout is short enough that if you do a click-and-hold for drag-and-drop, the cursor will move as expected. If a touch started outside a software button area, we can now use the whole touchpad for movement. And finally, a few fixes to avoid erroneous click events - we'd sometimes get the software button wrong if the event sequence is off.

Another change changed the behaviour of the touchpad when it is disabled through the "Synaptics Off" property. If you use syndaemon to disable the touchpad while typing, the buttons now work even when the touchpad is disabled. If you don't like touchpads at all and prefer to use the trackstick only, use Option "TouchpadOff" "1". This will disable everything but physical clicks on the touchpad.

On that note I'd also like to mention another touchpad bug that was fixed in the recent weeks: plenty of users reported synaptics having a finger stuck after suspend/resume or sometimes even after logging in. This was an elusive bug and finally tracked down to a mishandling of SYN_DROPPED events in synaptics 1.7 and libevdev. I won't provide a fix for synaptics 1.7 but we've fixed libevdev - please use synaptics 1.8 RC1 or later and libevdev 1.1 RC1 or later.

The kernel fix is more complicated. The problem is that while the touchpads all have a unique PNPID, that ID is not easily accessible because it shows up on a device that is not a parent of the device we see in the driver. Hence MatchPnPID doesn't apply. Matthew Garrett had a preliminary patch for this but it turned out to break some other use-case so we're back at square one at the moment. For now, I'm using a couple of udev rules together with the MatchTag configuration:

ATTR{[dmi/id]product_version}=="*T540*", ENV{ID_INPUT.tags}="T540"
and with the matching xorg.conf snippet:
Section "InputClass"
        Identifier "Lenovo T540 trackstick software button buttons"
        MatchTag "T540"
        Option "SecondarySoftButtonAreas" "3363 0 0 2280 2717 3362 0 2280"
EndSection
Oh btw, do you note the magic numbers here? This really should be in percent of the touchpad but we don't have the kernel patch. So for now I just ship the absolute numbers in the Fedora packages and glare pointedly at everyone who thinks that's a permanent solution.

Fedora users: everything is being built in rawhide and I have a F20 Copr that I'll keep pushing to once I get the various other bits for F20 sorted.

Monday, March 10, 2014

Using git - the next level

There's a million tutorials out there how to learn git. This isn't one of them. I'm going to assume that you learned git a while ago, you've been using it a bit and you're generally familiar with its principles. I'm going to show is a couple of things that improved my workflow. Chances are, it will improve yours too. This isn't a tutorial though. I'm just pointing you in the direction of things, you'll have to learn how to use them yourself.

Use tig

Seriously. Don't tell me you use gitk or git log is good enough for you. Use tig. tig is to git log what mutt is to mail(1). It has been the source of the biggest efficiency increase for me. Screenshots don't do it justice because the selling point is that it is interactive. But anyway, here are some official screenshots: tig blame shows you the file and the commits, you just need to select the line, hit enter and you see the actual commit. The main view by default shows you tags, branch names, remote branch names, etc. So not only do you immediately know which branch you're on, you will see local branches that have been merged, tags that have been applied, etc. It gives you an awareness that git log doesn't. Do yourself a favour, install it, use it for a day or two and I'm pretty sure you won't go back.

tig also supports custom configurations. Here is my $HOME/.tigrc:

bind generic X !git cherry-pick -x %(commit)
bind generic C !git cherry-pick %(commit)
bind generic R !git revert %(commit)
bind generic E !git format-patch -1 %(commit)
bind generic 0 !git checkout %(commit)
bind generic 9 !git checkout %(commit)~
bind generic A !git commit --amend -s
bind generic S !git show %(commit)
So with a couple of key strokes I can cherry-pick, export patches, revert, check out a single tree, etc. Especially cherry-picking is extremely efficient: check out the target branch, run "tig master", then simply select each commit, it "C" or "X" and done.

Use branches

Anytime it takes you more than 5 minutes to fix an issue, create a new branch. I'm getting torn between multiple things all the time. I may spend a day or two on one bug, then it's back to another, unrelated issue. With the review requirements on some projects I may have multiple patches waiting for feedback, but I can't push them yet. Hence - a branch for each feature/bugfix. master is reserved for patches that can be pushed immediately.

This approach becomes particularly useful for fixes that may need some extra refacturing. You start on a feature-based branch, but halfway through realise you need a few extra patches to refactor things. Those are easy to review so you send them out to gather reviews, then cherry-pick them to master and push. Back to your feature branch, rebase and you're done - you've managed two separate streams of fixes without interference. And most importantly, you got rid of a few patches that you'd otherwise have to carry in your feature branch.

Of course, it takes a while to get used this and it takes discipline. It took me a few times before I really managed to always work like this but the general rule for me is now: if I'm hacking on the master branch, something is off. Remember: there's no real limit to how many branches you can create - just make sure you clean them up when you're done to keep things easy for your brain.

Use the branch names to help you. You can rename branches (git branch -m), so I tend to name anything that's a bigger rewrite with "wip/somefeature" whereas normal bug fixes go on branches with normal names. And because I rebase local feature branches it doesn't matter what I name them anyway, the branches are deleted once I merge them. Branches where I do care about the branch history (i.e. those I pull them into master with a merge commit) I rename before pulling to get rid of the "wip" prefix.

Use branch descriptions

Hands up if you have a "devel" branch from 4 months ago. Hands up if you still remember what the purpose of that branch was. Right, I didn't think so. git branch --edit-description fires up an editor and lets you add a description for the branch. Sometimes a single sentence is enough to refresh your memory. Most importantly: when you task-switch to a different feature, edit the description to note where you left off, what the plan was, etc. This reduces the time to get back to work. git config branch.<branchname>.description shows you the description for the matching branch.

I even have a git hook to nag me when I check out a branch without a description. Note that branch descriptions are local only, they are not pushed to the remote.

Amend and rebase until the cows come home

The general rule: what is committed, doesn't get lost. At least not easily, it is still in the git reflog. So commit when you think you're done. Then review, test, add, and git commit --amend. That typo you made in line 4 - edit and amend. I have shell aliases for amend, rbs (git rebase -i) and rbc (git rebase --continue), and almost every commit goes through at least 3 amends (usually one because I missed something, one for that typo, one for commit log message editing). Importantly: it doesn't matter how often you amend. Really. This is local only, no-one cares. The important thing is that you get to a good patch set, not that you get there with one commit.

git commit --amend only modifies the last commit, to go back and edit the past, you need to rebase. So, you need to

Learn how to rebase

Not just the normal git rebase, the tutorials cover that. Make sure you know how to use git rebase --interactive. Make sure you know how to change the ordering of a commit, how to delete commits, how to abort a rebase. Make sure you know how to squash two commits together and what the difference is between squash and fixup. I'm not going to write a tutorial on that, because you can find the documentation is easy enough to find. Simply take this as a hint that the time you spend learning how to rebase pays off. Also, you may find git squash interesting.

And remember: even if a rebase goes bad, the previous state is still in the reflog. Which brings me to:

Learn how to use the reflog

The git reflog is the list of changes in reverse chronological order of how they were applied to the repository, regardless what branch you're on. So HEAD@{0} is always "whatever we have now", HEAD@{1} is always "the repository before the last command". This doesn't just mean commits, it remembers any change. So if you switch from branch A to branch B, commit something, then switch to branch C, HEAD@{3} is A. git reflog helpfully annotates everything with the type, so you know what actually happened. So for example, if you accidentally dropped a patch during a rebase, you can look at the reflog, figure out when the rebase started. Then you either reset to that commit, or you just tig it and cherry-pick the missing commits back onto the current branch. Create yourself up with a test git repository and learn how to do exactly that now, it'll save you some time in the future.

Note that the reflog is local only. And remember, if it hasn't been committed, it's not in the reflog.

Use a git push hook

Repeat after me: echo make > .git/hooks/pre-push. And no more embarrassment for pushing patches that don't compile. I've made that mistake too many times, so now I even use my own git patch-set command that will run a hook for me when I'm generating a patch set to send to a list. You might want to make the hooks executable btw.

Monday, February 10, 2014

Making sense of backtraces with addr2line

When the X server crashes it prints a backtrace to the log file. This backtrace looks something like this:

(EE) Backtrace:
(EE) 0: /usr/bin/Xorg (OsLookupColor+0x129) [0x473759]
(EE) 1: /lib64/libpthread.so.0 (__restore_rt+0x0) [0x3cd140f74f]
(EE) 2: /lib64/libc.so.6 (__select_nocancel+0xa) [0x3cd08ec78a]
(EE) 3: /usr/bin/Xorg (WaitForSomething+0x1ac) [0x46a8fc]
(EE) 4: /usr/bin/Xorg (SendErrorToClient+0x111) [0x43a091]
(EE) 5: /usr/bin/Xorg (_init+0x3b0a) [0x42c00a]
(EE) 6: /lib64/libc.so.6 (__libc_start_main+0xf5) [0x3cd0821d65]
(EE) 7: /usr/bin/Xorg (_start+0x29) [0x428c35]
(EE) 8: ? (?+0x29) [0x29]
This is a forced backtrace from the current F20 X Server package, generated by killall -11 Xorg. There is not a lot of human-readable information but you can see the call stack, and you can even recognise some internal functions. Now, in Fedora we compile with libunwind which gives us relatively good backtraces. Without libunwind, your backtrace may look like this:
(EE) 
(EE) Backtrace:
(EE) 0: /opt/xorg/bin/Xorg (xorg_backtrace+0xb5) [0x484989]
(EE) 1: /opt/xorg/bin/Xorg (0x400000+0x8d1a4) [0x48d1a4]
(EE) 2: /lib64/libpthread.so.0 (0x3cd1400000+0xf750) [0x3cd140f750]
(EE) 3: /lib64/libc.so.6 (__select+0x33) [0x3cd08ec7b3]
(EE) 4: /opt/xorg/bin/Xorg (WaitForSomething+0x3dd) [0x491a45]
(EE) 5: /opt/xorg/bin/Xorg (0x400000+0x3561b) [0x43561b]
(EE) 6: /opt/xorg/bin/Xorg (0x400000+0x43761) [0x443761]
(EE) 7: /opt/xorg/bin/Xorg (0x400000+0x9baa8) [0x49baa8]
(EE) 8: /lib64/libc.so.6 (__libc_start_main+0xf5) [0x3cd0821d65]
(EE) 9: /opt/xorg/bin/Xorg (0x400000+0x25df9) [0x425df9]
So, even less information and it certainly makes it hard to figure out where to even get started. Luckily there is a tool to get some useful info out of that: eu-addr2line. All you need is to install the debuginfo package for the crashing program. Then it's just a matter of copying addresses.
$ eu-addr2line -e /opt/xorg/bin/Xorg 0x48d1a4
/home/whot/xorg/xserver/os/osinit.c:132
Alright, this is useful now, I can download the source package and check where it actually goes wrong. But wait - it gets even better. Let's say you have a driver module in the callstack:
(EE) Backtrace:
(EE) 0: /opt/xorg/bin/Xorg (xorg_backtrace+0xb5) [0x484989]
(EE) 1: /opt/xorg/bin/Xorg (0x400000+0x8d1a4) [0x48d1a4]
(EE) 2: /lib64/libpthread.so.0 (0x3cd1400000+0xf750) [0x3cd140f750]
(EE) 3: /opt/xorg/lib/libinput.so.0 (libinput_dispatch+0x19) [0x7ffff1e51593]
(EE) 4: /opt/xorg/lib/xorg/modules/input/libinput_drv.so (0x7ffff205b000+0x2a12) [0x7ffff205da12]
(EE) 5: /opt/xorg/bin/Xorg (xf86Wakeup+0x1b1) [0x4af069]
(EE) 6: /opt/xorg/bin/Xorg (WakeupHandler+0x83) [0x444483]
(EE) 7: /opt/xorg/bin/Xorg (WaitForSomething+0x3fe) [0x491a66]
(EE) 8: /opt/xorg/bin/Xorg (0x400000+0x3561b) [0x43561b]
(EE) 9: /opt/xorg/bin/Xorg (0x400000+0x43761) [0x443761]
(EE) 10: /opt/xorg/bin/Xorg (0x400000+0x9baa8) [0x49baa8]
(EE) 11: /lib64/libc.so.6 (__libc_start_main+0xf5) [0x3cd0821d65]
(EE) 12: /opt/xorg/bin/Xorg (0x400000+0x25df9) [0x425df9]
You can see that we have a xf86libinput driver (libinput_drv.so) which in turn loadsd libinput.so. You can debug the crash the same way now, just change the addr2line argument:
$ eu-addr2line -e /opt/xorg/lib/libinput.so.0 libinput_dispatch+0x19 
/home/whot/code/libinput/src/libinput.c:603
Having this information of course doesn't mean you can fix any bug. But when you're reporting a bug it can be invaluable. If I have access to the same rpms that you're running it's possible to look up the context of the crash in the source. Or, even better, since you already have access to those you can make debugging a lot easier by attaching the required bits and pieces to a bug report. Seeing a bug report where a reporter already narrowed down where it crashes makes it a lot easier than guessing based on hex numbers what went wrong.

Friday, December 13, 2013

Lenovo T440 touchpad button configuration

Update March 19 2014: this post is outdated, please read X.Org synaptics support for the Lenovo T440, T540, X240, Helix, Yoga, X1 Carbon instead.

The T440 has a rather unusual touchpad with the buttons painted on top of the touchpad rather than the bottom. In addition, the separate set of buttons for the trackstick have gone the way of the dodo. Moving the software-emulated buttons up on the touchpad is obviously quite important for trackstick users but it throws up a bunch of problems. There are some limitations with the current synaptics X.Org driver: we can only have one region each designated for the right and the middle button. The rest of the touchpad is a left button click. In the case of the T440, the default Windows config has a right button up the top and another one at the bottom of the touchpad. An ASCII-art of that would look like this:

+----------------------------+
| LLLLLLLLLL MMMMM RRRRRRRRR |
|                            |
|                            |
|                            |
|                            |
|                            |
|                            |
| LLLLLLLL          RRRRRRRR |
+----------------------------+
We simply can't do that at the moment, best we can is split the touchpad so that the whole right side is a right-click and a strip in the middle that is a middle click. So the best we can do is:
+----------------------------+
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
+----------------------------+
I'm working on a solution for the proper config, but for now you'll have to be content with this.

The easiest approach for local configuration is a new InputClass section in the form:

Section "InputClass"
    Identifier "t440 top buttons"
    MatchDriver "synaptics"
    #                         right btn|middle btn
    Option "SoftButtonAreas" "60% 0 0 0 40% 60% 0 0"
EndSection
Drop that into /etc/X11/xorg.conf.d/99-t440-synaptics.conf and you're good to go.

The problem is finding a generic solution to this that we can ship in a distribution. That requires a two-step progress. The touchpads look the same as all others, the only differentiator we have is the DMI information on the box. We can't check that in the xorg.conf snippets yet (Daniel Martin is working on a MatchDMI tag, but it won't happen until server 1.16). For now, we need a udev rule to help the xserver.

ACTION!="add|change", GOTO="touchpad_quirks_end"
KERNEL!="event*", GOTO="touchpad_quirks_end"
ENV{ID_INPUT_TOUCHPAD}!="1", GOTO="touchpad_quirks_end"

ATTR{[dmi/id]product_version}=="*T440*", \
  ENV{ID_INPUT.tags}="touchpad_softbutton_top"

LABEL="touchpad_quirks_end"
If our product matches T440, we tag the touchpad, and that tag is something we can match against. Our revised InputClass section now looks like this:
Section "InputClass"
    Identifier "t440 top buttons"
    MatchDriver "synaptics"
    MatchTag "touchpad_softbutton_top"
    Option "SoftButtonAreas" "60% 0 0 0 40% 60% 0 0"
EndSection
I've pushed this configuration into Fedora now (rawhide, F20, F19), let's see what the feedback is. Having the whole right-side of the touchpad work as right button may cause a few issues so this is one change I may have to revert in the future.