War stories. Part 2

@CesantaHQ

by Sergey Lyubka, Cesanta CTO, for Dublin C/C++ User Group

Broken downloads

• A problem was reported that the HTTP sever does not serve certain files correctly
• Debugging in a browser revelead that the Content-Length header was too large

#if defined(__MINGW32__)
typedef struct stat cs_stat_t;
#else
typedef struct _stati64 cs_stat_t;
#endif


  int mg_stat(const char *path, cs_stat_t *st) {
     ....
     return _wstati64(wpath, (struct _stati64 *) st);
  }

  ...

  mg_printf(conn, "Content-Length: %Iu\r\n", st->st_size);

Broken downloads

• Bottom line: fixed next day
• Unit test added
• Disappointed users

A hilarious debugging session

• Two terminals
  • First is used to edit the code in vim
  • Second is used to compile & run
• Typical printf - debugging
• Server segfaults on fragmented websocket frame

$ make clean all COPTS="-O0 -g" && ./srv
...
Staring on port 7021
...
Segmentation fault: 11

static int deliver(struct mg_connection *c) {
   ...
   if (!is_first_fragment(p[0])) {
     // defragmentation code ...
     ....
   }
   ...
}

Terminal 1

Terminal 2

A hilarious debugging session

• Okay let's put some useful printfs into the defragmentation code

$ make clean all COPTS="-O0 -g" && ./srv
...
Staring on port 7021
...
Segmentation fault: 11

static int deliver(struct mg_connection *c) {
   ...
   if (!is_first_fragment(p[0])) {
     printf("boyz, tornado is coming\n"); // <--
     // defragmentation code ...
     ....
   }
   ...
}

Terminal 1

Terminal 2

• Recompiled and run - a message does not appear though

A hilarious debugging session

• Okay moved the printf to the function start

$ make clean all COPTS="-O0 -g" && ./srv
...
Staring on port 7021
...
Segmentation fault: 11

static int deliver(struct mg_connection *c) {
   printf("boyz, tornado is coming\n");  // <--
   ...
   if (!is_first_fragment(p[0])) {
     ....
   }
   ...
}

Terminal 1

Terminal 2

• No change ...

A hilarious debugging session

• Moved printf to main()

$ make clean all COPTS="-O0 -g" && ./srv
...
Staring on port 7021
...
Segmentation fault: 11

int main(int argc, const char *argv[]) {
   printf("boyz, tornado is coming\n");  // <--
   ...
}

Terminal 1

Terminal 2

• No change !!
• WTF?

A hilarious debugging session

• Bottom line: about 2 hours wasted
• Disassembling revealed no signs of printf
• Suspected filesystem to not saving a file
• Suspected compiler to not to recompile
• Suspected nihilists to hack vim

A weird compilation error

static void ev_handler(struct mg_connection *c,
                       int ev,
                       void *ev_data) {
  ...
  c->flags |= WEB_TTY;
  ...
}

Before:

After:

static void ev_handler(struct mg_connection *c,
                       int ev,
                       void *ev_data) {
  ...
  c->flags |= WEB_TTY;
  ...
  printf("%p connected\n", c);  // <-- Added
  ...
}

error: format specifies type 'void *' but the argument
       has type '<dependent type>' [-Werror,-Wformat]
testutil.c:19:32: error: use of undeclared identifier 'c'
      printf("%p connected\n", c);
                               ^

Results in compilation error:

A weird compilation error

• Bottom line: about 20 minutes wasted
• Renaming a variable helped
• Hexdump revealed the reason

Thank you!

contact me at

sergey.lyubka@cesanta.com