Debugging Subversion ¶

Debugging the DAV server ¶

'mod_dav_svn.so' contains the main Subversion server logic; it runs as a module within mod_dav, which runs as a module within httpd. If httpd is probably using dynamic shared modules, you might need to set breakpoint pending on (in ~/.gdbinit) before setting breakpoints in mod_dav_svn would be possible. Alternatively, you may start httpd, interrupt it, set your breakpoint, and continue:

   % gdb httpd
   (gdb) run -X
   ^C
   (gdb) break some_func_in_mod_dav_svn
   (gdb) continue

The -X switch is equivalent to -DONE_PROCESS and -DNO_DETACH, which ensure that httpd runs as a single thread and remains attached to the tty, respectively. As soon as it starts, it sits and waits for requests; that's when you hit control-C and set your breakpoint.

You'll probably want to watch Apache's run-time logs

   /usr/local/apache2/logs/error_log
   /usr/local/apache2/logs/access_log

to help determine what might be going wrong and where to set breakpoints.

Alternatively, running ./subversion/tests/cmdline/davautocheck.sh --gdb in a working copy will start httpd using the mod_dav_svn in that working copy. You can then run individual Python tests against that: ./basic_tests.py --url=http://localhost:3691/.

Debugging the ra_svn client and server, on Unix ¶

Bugs in ra_svn usually manifest themselves with one of the following cryptic error messages:

  svn: Malformed network data
  svn: Connection closed unexpectedly

(The first message can also mean the data stream was corrupted in tunnel mode by user dotfiles or hook scripts; see issue #1145.) The first message generally means you to have to debug the client; the second message generally means you have to debug the server.

It is easiest to debug ra_svn using a build with --disable-shared --enable-maintainer-mode. With the latter option, the error message will tell you exactly what line to set a breakpoint at; otherwise, look up the line number at the end of marshal.c:vparse_tuple() where it returns the "Malformed network data" error.

To debug the client, simply pull it up in gdb, set a breakpoint, and run the offending command:

  % gdb svn
  (gdb) break marshal.c:NNN
  (gdb) run ARGS
  Breakpoint 1, vparse_tuple (list=___, pool=___, fmt=___, 
    ap=___) at subversion/libsvn_ra_svn/marshal.c:NNN
  NNN                                 "Malformed network data");

There are several bits of useful information:

• A backtrace will tell you exactly what protocol exchange is failing.

• "print *conn" will show you the connection buffer. read_buf, read_ptr, and read_end represent the read buffer, which can show you the data the marshaller is looking at. (Since read_buf isn't generally 0-terminated at read_end, be careful of falsely assuming that there's garbage data in the buffer.)

• The format string determines what the marshaller was expecting to see.

To debug the server in daemon mode, pull it up in gdb, set a breakpoint (usually a "Connection closed unexpectedly" error on the client indicates a "Malformed network data" error on the server, although it can also indicate a core dump), and run it with the "-X" option to serve a single connection:

  % gdb svnserve
  (gdb) break marshal.c:NNN
  (gdb) run -X

Then run the offending client command. From there, it's just like debugging the client.

Debugging the server in tunnel mode is more of a pain. You'll need to stick something like "{ int x = 1; while (x); }" near the top of svnserve's main() and put the resulting svnserve in the user path on the server. Then start an operation, gdb attach the process on the server, "set x = 0", and step through the code as desired.

Network tracing with Wireshark ¶

Use Wireshark (formerly known as "Ethereal") to eavesdrop on the conversation.

First, make sure that between captures within the same wireshark session, you hit Clear, otherwise filters from one capture (say, an HTTP capture) might interfere with others (say, an ra_svn capture).

Assuming you're cleared, then:

1. Pull down the Capture menu, and choose Capture Filters.

2. If debugging the http:// (WebDAV) protocol, then in the window that pops up, choose "HTTP TCP port (80)" (which should result in the filter string "tcp port http").

   If debugging the svn:// (ra_svn) protocol, then choose New, give the new filter a name (say, "ra_svn"), and type "tcp port 3690" into the filter string box.

   When done, click OK.

3. Again go to the Capture menu, this time choose Interfaces, and click Options next to the appropriate interface (probably you want interface "lo", for "loopback", assuming the server will run on the same machine as the client).

4. Turn off promiscuous mode by unchecking the appropriate checkbox.

5. Click the Start button in the lower right to start the capture.

6. Run your Subversion client.

7. Click the Stop icon (a red X over an ethernet interface card) when the operation is finished (or Capture->Stop should work). Now you have a capture. It looks like a huge list of lines.

8. Click on the Protocol column to sort.

9. Then, click on the first relevant line to select it; usually this is just the first line.

10. Right click, and choose Follow TCP Stream. You'll be presented with the request/response pairs of the Subversion client's HTTP conversion.

The above instructions are specific to the graphical version of Wireshark (version 0.99.6), and don't apply to the command-line version known as "tshark" (which corresponds to "tethereal", from back when Wireshark was called Ethereal).

Network tracing with socat ¶

Another alternative is to set up a logging proxy between the Subversion client and server. A simple way to do this is to use the socat program. For example, to log communication with an svnserve instance, run the following command:

socat -v TCP-LISTEN:9630,reuseaddr,fork TCP4:localhost:svn

Then run your svn commands using an URL base of svn://127.0.0.1:9630/; socat will forward the traffic from port 9630 to the normal svnserve port (3690), and will print all traffic in both directions to standard error, prefixing it with < and > signs to show the direction of the traffic.

To log readable HTTP from an encrypted https:// connection run a socat proxy that connects using TLS to the server:

socat -v TCP-LISTEN:9630,reuseaddr,fork OPENSSL:example.com:443

and then use that as a proxy for a plain http:// connection:

svn ls http://example.com/svn/repos/trunk \
    --config-option servers:global:http-proxy-host=localhost \
    --config-option servers:global:http-proxy-port=9630 \
    --config-option servers:global:http-compression=no

The socat proxy logs the plain HTTP while all the network traffic with the server is encrypted using TLS.

Network tracing with http-proxy ¶

If you're debugging a http client/server setup you can use a web debugging proxy like Charles or Fiddler.

Once you've setup such a proxy you'll need to configure Subversion to use the proxy. This can be done with the http-proxy-host and http-proxy-port parameters in the servers configuration file. You can also specify the proxy on the command line with these options --config-option 'servers:global:http-proxy-host=127.0.0.1' --config-option 'servers:global:http-proxy-port=8888'.