FcgidPassHeader Authorization

Das Problem von dem ich spreche ist der Dateiexport von Zotero, einem Programm das ich zur Verwaltung meiner Bibliographie verwende, nach BibTeX, dem Dateiformat das ich zur automatischen Erstellung der Referenzen und des Quellenverzeichnisses in meiner Dissertation verwende. Wenn nun im Titel von verwendeten Quellen oder in Namen von deren Autoren deutsche Umlaute oder diakritische Zeichen (wie z.B. französische Akzente) vorkommen, werden diese in der exportierten Datei unverändert wiedergegeben, weil Zotero die ausgegebene Datei per Vorgabe als UTF-8 kodiert. Diese Kodierung wird aber von BibTeX nicht richtig verarbeitet.

Für diese Problem gibt es zwei Lösungswege, wobei ich leider viel Zeit mit dem ersten verloren habe:

1. Man kann versuchen, BibTeX zur richtigen Verarbeitung der UTF-8 Datei zu bringen, bzw. einen dementsprechenden Ersatz für BibTeX einzusetzen. Diesbezüglich gibt es jede Menge Diskussionen im Internet, die die Hoffnung (leider) nicht schnell genug sterben lassen. Der Vollständigkeit halber sei hier beispielhaft auf bibtex8 verwiesen.
2. Viel einfacher ist es jedoch, Zotero dazu zu bringen, die kritischen Zeichen entsprechend der LaTeX Syntax auszugeben. Das schafft man aber nicht, indem man selbst die Einträge entsprechend ändert, sondern indem man die Dateikodierung für den Export ändert.
   
   Dazu muss man lediglich in den Einstellungen von Zotero in der Kategorie Export ein Häkchen bei “Option für Zeichenkodierung beim Export anzeigen” setzen. Beim nächsten BibTeX Export ist dann einfach “Westlich (ISO-8859-1)” aus der Liste auszuwählen und schon erstellt Zotero eine problemlos verwendbare BibTeX Datei.

--- old/custom.conf	2011-07-07 09:35:01.462829111 +0200
+++ new/custom.conf	2011-07-07 09:30:27.146258540 +0200
@@ -7,6 +7,7 @@
 [xdmcp]
 
 [greeter]
+IncludeAll=false
 
 [chooser]

Damit werden nicht oder zB nur per scp genutzte Benutzernamen ausgeblendet.

This article has multiple motivations. First, it is a follow-up to an article by Patrick Ohly (author of syncEvolution). As such it illustrates my own experience with his software, but also is meant as a way to show my gratitude and give kudos to him and all contributors of syncEvolution. Second, but this is somehow related to the aforementioned article, it should show that synchronisation [of PIM data] can work reliably and as such it opposes a wide spread opinion.

Scope

What I’ll try to show is how I’m successfully synchronising my notebook, my workstation (both Fedora 15) and my Android phone using the SyncML protocol.

Disclaimer: on the phone, I’m currently using a non-free piece of software from Synthesis AG for the only reason that the free Funambol client doesn’t play well with my self-signed SSL certificate (yet!?).

Server aka “cloud”

On the server-side, I’m using the eGroupware software. That means PHP with MySQL backend and a few cron scripts, hence quiet a “default” set-up and easily joining my WordPress. For installation and updates I’m using their CentOS 5 repository hosted at the openSUSE build service.

eGroupware exposes a SyncML interface, but also supports GroupDAV, CalDAV and CardDAV for those with different tastes and/or clients.

Desktops

Although syncEvolution is not intended to be limited to the GNOME Evolution software, that is what I’m using and the inter-play between the two works out of the box. To also work with the server, syncEvolution needs some tuning. After installation of syncevolution-gtk, I created a stub configuration with

$ syncevolution --configure --sync-property "username=MYEGROUPWAREUSER" memotoo

then renamed it to match my own service

$ cd ~/.config/syncevolution
$ mv default/peers/memotoo default/peers/fabianmoser

finally, I change the default setting to match my server by applying a patch (below)

$ patch -p1 < ~/syncevolution.patch

Here is the syncevolution.patch file (explanations below):

diff -u -r old/config.ini new/config.ini
--- old/config.ini	2011-07-06 09:45:01.952470598 +0200
+++ new/config.ini	2011-07-01 13:27:33.000000000 +0200
@@ -1,2 +1,2 @@
 # the peer which is used by default in some frontends, like the sync-UI
-# defaultPeer = 
+defaultPeer = fabianmoser
diff -u -r old/default/peers/fabianmoser/config.ini new/default/peers/fabianmoser/config.ini
--- old/default/peers/fabianmoser/config.ini	2011-07-06 09:48:16.758035192 +0200
+++ new/default/peers/fabianmoser/config.ini	2011-07-04 14:21:47.000000000 +0200
@@ -16,7 +16,7 @@
 # In the future this might be used to contact the peer
 # via one of several transports; right now, only the first
 # one is tried.
-syncURL = http://sync.memotoo.com/syncML
+syncURL = https://fabianmoser.at/egroupware/rpc.php
 
 # user name used for authorization with the SyncML server
 username = MYEGROUPWAREUSER
@@ -40,7 +40,7 @@
 
 # enables or disables the detailed (and sometimes slow) comparison
 # of database content before and after a sync session
-# printChanges = 1
+printChanges = 0
 
 # Controls automatic synchronization. Currently,
 # automatic synchronization is done by running
@@ -57,7 +57,7 @@
 # http - only via HTTP transport
 # obex-bt - only via Bluetooth transport
 # http,obex-bt - pick one of these
-# autoSync = 0
+autoSync = 1
 
 # This is the minimum number of seconds between two
 # synchronizations that has to pass before starting
@@ -72,7 +72,7 @@
 # synchronization when changes are detected (not
 # implemented yet, therefore it basically disables
 # automatic synchronization).
-# autoSyncInterval = 30M
+autoSyncInterval = 60M
 
 # An automatic sync will not be started unless the peer
 # has been available for this duration, specified in seconds
@@ -84,7 +84,7 @@
 # to predict how long connectivity be available in the
 # future, because it should better be available long
 # enough to complete the synchronization.
-# autoSyncDelay = 5M
+autoSyncDelay = 5M
 
 # During a slow sync, the SyncML server must match all items
 # of the client with its own items and detect which ones it
@@ -185,7 +185,7 @@
 # An arbitrary name for the peer referenced by this config.
 # Might be used by a GUI. The command line tool always uses the
 # the configuration name.
-# PeerName = 
+PeerName = fabianmoser
 
 # SyncML ID of our peer, empty if unknown; must be set only when
 # the peer is a SyncML client contacting us via HTTP.
@@ -221,7 +221,7 @@
 # 
 # SSL support when acting as HTTP server is implemented
 # by the HTTP server frontend, not with these properties.
-# SSLServerCertificates = /etc/ssl/certs/ca-certificates.crt:/etc/pki/tls/certs/ca-bundle.crt:/usr/share/ssl/certs/ca-bundle.crt
+SSLServerCertificates = /home/fabian/.local/share/ssl/certs/fabianmoser.at.crt
 
 # The client refuses to establish the connection unless
 # the server presents a valid certificate. Disabling this
@@ -238,7 +238,7 @@
 
 # The URL of a web page with further information about the server.
 # Used only by the GUI.
-WebURL = http://www.memotoo.com
+WebURL = http://fabianmoser.at
 
 # The URI of an icon representing the server graphically.
 # Should be a 48x48 pixmap or a SVG (preferred).
diff -u -r old/default/peers/fabianmoser/sources/addressbook/config.ini new/default/peers/fabianmoser/sources/addressbook/config.ini
--- old/default/peers/fabianmoser/sources/addressbook/config.ini	2011-07-06 09:48:16.758035192 +0200
+++ new/default/peers/fabianmoser/sources/addressbook/config.ini	2011-07-01 13:27:33.000000000 +0200
@@ -141,8 +141,8 @@
 #       xmlrpc:text/vcard:3.0
 #       xmlrpc:text/x-vcalendar:1.0
 #       xmlrpc:text/calendar:2.0
-type = addressbook
+type = addressbook:text/x-vcard
 
 # this is appended to the server's URL to identify the
 # server's database
-uri = con
+uri = ./contacts
diff -u -r old/default/peers/fabianmoser/sources/calendar/config.ini new/default/peers/fabianmoser/sources/calendar/config.ini
--- old/default/peers/fabianmoser/sources/calendar/config.ini	2011-07-06 09:48:16.758035192 +0200
+++ new/default/peers/fabianmoser/sources/calendar/config.ini	2011-07-01 13:27:33.000000000 +0200
@@ -141,8 +141,8 @@
 #       xmlrpc:text/vcard:3.0
 #       xmlrpc:text/x-vcalendar:1.0
 #       xmlrpc:text/calendar:2.0
-type = calendar
+type = calendar:text/calendar
 
 # this is appended to the server's URL to identify the
 # server's database
-uri = cal
+uri = ./calendar
diff -u -r old/default/peers/fabianmoser/sources/todo/config.ini new/default/peers/fabianmoser/sources/todo/config.ini
--- old/default/peers/fabianmoser/sources/todo/config.ini	2011-07-06 09:48:16.759035180 +0200
+++ new/default/peers/fabianmoser/sources/todo/config.ini	2011-07-01 12:02:16.000000000 +0200
@@ -145,4 +145,4 @@
 
 # this is appended to the server's URL to identify the
 # server's database
-uri = task
+uri = ./tasks
diff -u -r old/default/sources/addressbook/config.ini new/default/sources/addressbook/config.ini
--- old/default/sources/addressbook/config.ini	2011-07-06 09:45:01.953470586 +0200
+++ new/default/sources/addressbook/config.ini	2011-07-01 13:27:33.000000000 +0200
@@ -123,7 +123,7 @@
 #       xmlrpc:text/vcard:3.0
 #       xmlrpc:text/x-vcalendar:1.0
 #       xmlrpc:text/calendar:2.0
-type = addressbook
+type = addressbook:text/x-vcard
 
 # Picks one of backend data sources:
 # enter either the name or the full URL.
diff -u -r old/default/sources/calendar/config.ini new/default/sources/calendar/config.ini
--- old/default/sources/calendar/config.ini	2011-07-06 09:45:01.953470586 +0200
+++ new/default/sources/calendar/config.ini	2011-07-01 13:27:33.000000000 +0200
@@ -123,7 +123,7 @@
 #       xmlrpc:text/vcard:3.0
 #       xmlrpc:text/x-vcalendar:1.0
 #       xmlrpc:text/calendar:2.0
-type = calendar
+type = calendar:text/calendar
 
 # Picks one of backend data sources:
 # enter either the name or the full URL.

What this does is setting the URL and configuring auto-sync for every 60 minutes. It tells syncEvolution that it should accept my SSL certificate. Finally, it changes the types and uris of the different SyncML sources, namely calendar, tasks and contacts, to match eGroupware.

Phone

The Synthesis client must again be pointed to the correct URL, i.e. the rpc.php file of the eGroupware installation. Finally, the options must be changed such that the client accepts the self-signed certificate. If necessary, you can configure to which Android storage your PIM data should be synchronised.

Conclusion

I hope I could give a credible example of how synchronisation in my own little private “cloud” works. If this helps someone to create a similar set-up, even better.

The only disadvantage I see at the moment is, that I have to rely on a non-free client on the phone. Maybe other SyncML clients will at some point be able to cope with self-signed certificates. Or maybe there will be free and reliable CalDAV & CardDAV clients for Android?

Gernot Walzl wrote a “nasty workaround” (quoting himself) in the form of the following script:

#!/bin/sh
 
# nvidia_dpms_fix.sh
# 2011-06-12
# by Gernot WALZL
 
# nasty workaround for nvidia drivers to resume from dpms off/suspend
 
export DISPLAY=:0
 
getXauthority () {
  export XAUTHORITY=$(ps -C X -f | grep "$DISPLAY" \
    | sed -n 's/.* -auth \([^ ]*\).*/\1/p')
}
 
handle_line () {
  if echo "$1" | grep -e "(EE) NVIDIA.* DisplayPort link training failed" \
      > /dev/null; then
    getXauthority
    xset dpms force on
  fi
}
 
tail --follow=name /var/log/Xorg.0.log --retry --lines=1 2> /dev/null \
  | while read line; do
    handle_line "$line"
  done

To actually make this do its job, I copied it into /usr/local/bin, made it executable and created an autostart entry for GNOME 3 by saving the following lines under ~/.config/autostart/nvidia_dpms_fix.sh.desktop

[Desktop Entry]
Type=Application
Exec=/usr/local/bin/nvidia_dpms_fix.sh
Hidden=false
X-GNOME-Autostart-enabled=true
Name=NVidia DPMS fix
Comment=Wakes up the display properly

I believe the problem that is being worked around is indicated by a line in the Xorg.0.log:

(WW) NVIDIA(GPU-0): AUO (DFP-3): Failed to set DisplayPort power state

Although I could only test this on my machine (HP EliteBook 8440p), with some luck this works for all cases where a similar log message is found.

Source first, discussion later:

package main
 
import (
	"fmt"
	"http"
	"log"
	"time"
)
 
func main() {
	http.HandleFunc("/test/", feed)
	if err := http.ListenAndServe("127.0.0.1:8080", nil); err != nil {
		log.Fatal(err)
	}
}
 
func feed(w http.ResponseWriter, r *http.Request) {
	ch := time.Tick(1000000000)
	fmt.Fprintf(w, "Goconntest welcomes you! Your Id: %s\n", r.URL.String())
	w.(http.Flusher).Flush()
	for N := 1; ; N++ {
		<-ch
		n, err := fmt.Fprintf(w, "Chunk %d for id %s\n", N, r.URL.String())
		if err != nil {
			log.Print(err)
			break
		} else if n == 0 {
			log.Print("Nothing written")
			break
		}
		w.(http.Flusher).Flush()
	}
}

Of course this could be stripped down by ignoring (i.e. not handling) errors, but IMHO this doesn’t fall behind the Erlang implementation in terms of beauty; well, my sense of “beauty” anyway.

You might have noted, that I’m not using the time.Sleep function, but instead a seemingly “artificial” construct with a time.Ticker channel. This internally allows for a multiplexing of the Goroutines spawned by the http module to only a few system threads, scheduling the network operations via epoll.

Update 2011-05-23: Removed usage of ChunkedWriter from Go source according to a hint from Brad Fitzpatrick.

Update 2011-05-25: The flushing by w.(http.Flusher).Flush() is necessary to make sure that the chunks get written in-time. Thanks to Brad Fitzpatrick for pointing this out.

For the client side, I used Richard’s floodtest.erl module, only removing the {version, 1.1} line from the http:request, because that option doesn’t seem to be supported by Erlang R14B.

The result from my Intel(R) Core(TM) i7 CPU M 620 @ 2.67GHz:

The RSS usage of goconntest_web converges at 321MB i.e. roughly 32KB per connection (13KB less than the ad-hoc Erlang implementation). Also note, that Go indeed spawned only 4 threads on this quad-core machine to handle the 10k open connections.

The only disappointment was the rather high CPU usage. Although I’m aware that the screen-shot is far from any acceptable benchmark measure, still 1.5h CPU time seems to be more than “practically nothing” Richard reported for the Erlang implementation.

Update 2011-05-25: While testing the new code above, I logged the process statistics with pidstat and then made the following plot (Python script, data file):

Mind the logarithmic time scale! The dashed line indicates the ramp-up of opened connections to 10k (theoretical, not measured). The colorful background is a stacked plot of the per-thread CPU usage with one color per kernel thread-id. Obviously Go spawns more threads than there are cores. Finally, the thick black line is the resident memory usage, which seems to converge much later than the CPU usage.

Thanks for reading. Maybe, if the App Engine Channels API takes long enough, I will continue my investigations.

Once a writing process is identified, lsof can tell you which files are actually written. Depending on what you want to achieve, you can then continue to relocate that file to a different disk (e.g. if you have two disks and want one of them to sleep most of the time) or to memory (using tmpfs). The latter obviously means that the written data will be lost at the next reboot, but sometimes this is perfectly fine e.g. for the files inside /tmp. Therefore, it is generally a good idea to move that directory into volatile memory following the instructions on the openSUSE Wiki.

All good and well if you actually manage to relocate or move a file. Often file locations are a matter of configuration and otherwise you can help yourself with dynamic links. But there was one very special file on my openSUSE 11.4 installation that withstood all my assaults for quiet some time. I’m speaking of a beast called .xsession-errors residing in your $HOME directory. Created by KDM, one would expect to be able to configure the location of that file. Indeed, there is a configuration option called ClientLogFile specifically for that purpose. Unfortunately this is only the first and easier of two necessary steps:

1. As root open your /usr/share/kde4/config/kdm/kdmrc, go to a section labelled [X-:0-Core] (there may be multiple of those, but don’t worry and just pick the last one) and add the following line:

   ClientLogFile=../../tmp/xsession-errors-%u

   This will move the file into the /tmp directory. Mind the path relative to $HOME.

2. Now the hidden piece: again as root open your /etc/X11/xdm/Xsession and make the following changes (I deliberately use the patch syntax here i.e. remove lines with a minus sign and add those with a plus sign):

   --- Xsession.old        2011-05-01 19:46:40.000000000 +0200
   +++ Xsession    2011-05-02 22:18:39.000000000 +0200
   @@ -123,8 +123,8 @@
        # GDM seems to handle this its self
        test -z "$GDMSESSION" || break
    
   -    # Once if KDM does handle this its self
   -    #test -z "$KDMSESSION" || break
   +    # KDM handles this itself
   +    test -z "$KDE_SESSION_VERSION" || break
    
        # Avoid bad symbolic links
        case "$errfile" in

Done; logout, restart KDM, re-login and check if the xsession-errors* exists at the new location. If so, remove your old one and cheer to a long living SSD. Only, of course, if the new location is not on your SSD, but e.g. in memory. It doesn’t hurt to re-check with pidstat.

-XX:-UseCompressedOops

to your eclipse.ini.

1. Prepare a partition on the USB stick and/or make sure there is enough space on it (it must be slightly larger than the ISO image).
2. Make sure it is flagged bootable. ( fdisk -l /dev/sdX is your friend)
3. Remember the name of the USB partition you’re going to use or if unlabelled, label it.
4. Loop-mount the ISO image using something like mount -o loop /path/to/iso /mnt/loop
5. Copy the content of the ISO over to the USB
6. Install GRUB on the USB by issuing grub-install --no-floppy --root-directory=/mnt/usb /dev/sdX
7. Create a /mnt/usb/boot/grub/grub.cfg with the following content

   menuentry "Fedora Live" {
    linux /isolinux/vmlinuz0 root=live:LABEL=XYZ rootfstype=auto ro liveimg quiet  rhgb rd_NO_LUKS rd_NO_MD rd_NO_DM
    initrd /isolinux/initrd0.img
   }
    
   menuentry "Fedora Live (Basic Video)" {
    linux /isolinux/vmlinuz0 root=live:LABEL=XYZ rootfstype=auto ro liveimg quiet  rhgb rd_NO_LUKS rd_NO_MD rd_NO_DM xdriver=vesa nomodeset
    initrd /isolinux/initrd0.img
   }

   where XYZ must be replaced by the actual partition name of your USB partition.

8. Unmount and boot

Here are the slightly modified steps:

1. Download the ISO image
2. Unmount USB
3. Delete partitions on the USB
4. Create one primary partition of type 0x0c (Win 95 FAT LBA) of size ??.
5. Make it bootable
6. Use mkfs.vfat -n USB /dev/sdX1 to create the filesystem
7. Make sure syslinux, and isomd5sum are installed
8. Use livecd-iso-to-disk --overlay-size-mb 512 /path/to/iso /dev/sdX1
9. Sync

Good luck!