<div dir="ltr">Hi,<div><br></div><div style>Unfortunately, I didn't find the time to test with DRBD 8.4.2 again so far.</div><div style>But I sadly encountered the same bug again in DRBD 8.4.3 during our Ganeti stress tests. We have a DRBD volume on top of two LV's (one for the data, the other for metadata). Xen uses this volume for a VM which gets migrated to the secondary node in our test. Therefore, Ganeti switches the DRBD volume to dual-primary mode (on the secondary node), promotes the secondary node to primary node and tells Xen to migrate the running instance to the (old) secondary node. Unfortunately, somewhere in the process DRBD looses synchronization with a ProtocolError and fails to reconnect, as this would create a split-brain situation.</div>
<div style><br></div><div style>So, as cool as it would have been, we won't be able to take advantage of the on-the-fly changing into dual-master mode and back. Instead, we will continue as with DRBD 8.3.x and detach the volume, change to dual-master mode, reattach and resync, and then do the same reversed after the instance migration...</div>
<div style><br></div><div style>Thanks,</div><div style>Thomas</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, May 6, 2013 at 8:22 AM, Thomas Thrainer <span dir="ltr"><<a href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I'm was running the vanilla Wheezy kernel for my DRBD 8.4.3 tests. If I get to it, I'll try DRBD 8.4.2 again with that kernel.</div>
<div>It would still be interesting if this issues is/was already known and "officially" fixed in DRBD 8.4.3 (the release notes don't mention it), as I can't quite test all possible combinations between kernels and DRBD versions.</div>
<div><br></div><div>Cheers,</div><div>Thomas</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, May 3, 2013 at 6:24 PM, Louis <span dir="ltr"><<a href="mailto:mailtolouis2020-forum@yahoo.com" target="_blank">mailtolouis2020-forum@yahoo.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:10pt;font-family:arial,helvetica,sans-serif"><div>Hi Thomas,</div><div><br></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif">
Why don't you re-test DRBD 8.4.2 using Wheezy (Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64 GNU/Linux) ? If the error show again, that could be the answer that the problem is fixed in 8.4.3.</div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif">
<br></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif"><br><span></span></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif">
<span>Regards</span></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif"><span>Louis</span></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif">
<br><blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;margin-top:5px;padding-left:5px"> <div style="font-family:arial,helvetica,sans-serif;font-size:10pt"> <div style="font-family:times new roman,new york,times,serif;font-size:12pt">
<div dir="ltr"> <hr size="1"> <font face="Arial"> <b><span style="font-weight:bold">From:</span></b> Thomas Thrainer <<a href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a>><br> <b><span style="font-weight:bold">To:</span></b> Louis <<a href="mailto:mailtolouis2020-forum@yahoo.com" target="_blank">mailtolouis2020-forum@yahoo.com</a>> <br>
<b><span style="font-weight:bold">Cc:</span></b> Luca
Fornasari <<a href="mailto:luca.fornasari@gmail.com" target="_blank">luca.fornasari@gmail.com</a>>; DRBD <<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a>> <br> <b><span style="font-weight:bold">Sent:</span></b> Friday, May 3, 2013 2:14 PM<div>
<div><br> <b><span style="font-weight:bold">Subject:</span></b> Re: [DRBD-user] Enabling two primaries break connection<br> </div></div></font> </div><div><div> <div><br><div><div dir="ltr">Hi,<div><br>
</div><div>I just re-tested using Wheezy (Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64 GNU/Linux) and DRBD 8.4.3.</div><div>I couldn't reproduce the original bug (having one side writing, and toggling the other side between secondary/primary). However, if I try to write on the "toggling" side to the disk when in primary mode, I could still produce ProtocolError's.</div>
<div>As that's (fortunately) not required by Ganeti, we'll can probably ignore this (although I will test some more of our use-cases with DRBD 8.4.3).</div><div><br></div><div>Can somebody confirm that the original behavior was a bug in DRBD 8.4.2 which was fixed in DRBD 8.4.3? Or could interactions with the kernel be the problem?</div>
<div><br></div><div>Thanks,</div><div>Thomas</div></div><div><br><br><div>On Fri, May 3, 2013 at 2:56 PM, Louis <span dir="ltr"><<a rel="nofollow" href="mailto:mailtolouis2020-forum@yahoo.com" target="_blank">mailtolouis2020-forum@yahoo.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:10pt;font-family:arial,helvetica,sans-serif"><div><br><span></span></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif">
<br><span></span></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif"><span>Linux intel1 3.5.0-23-generic #35~precise1-Ubuntu SMP</span></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif">
<br><span></span></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif"><span></span></div><div><br><blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;margin-top:5px;padding-left:5px">
<div style="font-family:arial,helvetica,sans-serif;font-size:10pt"> <div style="font-family:times new roman,new york,times,serif;font-size:12pt"> <div dir="ltr"> <hr size="1"> <font face="Arial"> <b><span style="font-weight:bold">From:</span></b> Luca Fornasari <<a rel="nofollow" href="mailto:luca.fornasari@gmail.com" target="_blank">luca.fornasari@gmail.com</a>><br>
<b><span style="font-weight:bold">To:</span></b> DRBD <<a rel="nofollow" href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a>> <br> <b><span style="font-weight:bold">Sent:</span></b> Friday, May 3, 2013 1:01 PM<div>
<div><br> <b><span style="font-weight:bold">Subject:</span></b> Re: [DRBD-user] Enabling two primaries break connection<br> </div></div></font> </div><div><div> <div><br><div><div dir="ltr">Hi Louis,<div>
<br></div><div>What kernel are yuo running?</div><div><br></div><div>Cheers,</div><div>Luca</div></div><div><br><br><div>On Fri, May 3, 2013 at 1:41 PM, Louis <span dir="ltr"><<a rel="nofollow" href="mailto:mailtolouis2020-forum@yahoo.com" target="_blank">mailtolouis2020-forum@yahoo.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:10pt;font-family:arial,helvetica,sans-serif"><div><span>Hi Thomas,</span></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif">
<br><span></span></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif">I just tested your scenario on DRBD 8.4.3. It seems fine, connection did not break.</div>
<div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif">While doing the DD, I did change primary / secondary on bob node few times, but the connection never break.</div>
<div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif"><br></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif">
The only thing I couldn't run this drbdadm --allow-two-primaries=yes <resource>, it give uknown command. But I do have allow-two-primaries setup in my drbd.conf, not sure that make any different?</div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif">
<br><span></span></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:arial,helvetica,sans-serif"><span></span></div><div>Regards</div><div>Louis<br><blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;margin-top:5px;padding-left:5px">
<div style="font-family:arial,helvetica,sans-serif;font-size:10pt"> <div style="font-family:times new roman,new york,times,serif;font-size:12pt"> <div dir="ltr"> <hr size="1"> <font face="Arial"> <b><span style="font-weight:bold">From:</span></b> Thomas Thrainer <<a rel="nofollow" href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a>><br>
<b><span style="font-weight:bold">To:</span></b> Luca Fornasari <<a rel="nofollow" href="mailto:luca.fornasari@gmail.com" target="_blank">luca.fornasari@gmail.com</a>> <br><b><span style="font-weight:bold">Cc:</span></b> <a rel="nofollow" href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a> <br>
<b><span style="font-weight:bold">Sent:</span></b> Friday, May 3, 2013 10:00 AM<br> <b><span style="font-weight:bold">Subject:</span></b> Re: [DRBD-user] Enabling two primaries break connection<br> </font> </div> <div><div>
<div><br><div><div dir="ltr">Hi,<div><br></div><div>Given the fact that I could reproduce the bug regularly, I'm pretty sure that this is indeed a DRBD bug. Unfortunately I didn't find a bugtracker on the DRBD site. Is there one? If not, what's the official way to report
bugs?</div>
<div><br></div><div>For the record, here are the steps to reproduce:</div><div> * create a device with primary on alice and secondary on bob</div><div> * on alice: while true; do dd if=/dev/zero of=/dev/drbd0; done</div>
<div> * on bob: drbdadm --allow-two-primaries=yes <resource>; drbdadm primary <resource></div><div> * you might have to drbdadm secondary <resource> on bob and then repeat the previous step in order to produce the bug</div>
<div><br></div><div>Log files are as mentioned in the first mail of the thread.</div><div><br></div><div>Cheers,</div><div>Thomas</div><div><br></div></div><div><br>
<br><div>On Tue, Apr 30, 2013 at 2:25 PM, Thomas Thrainer <span dir="ltr"><<a rel="nofollow" href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a>></span> wrote:<br><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Luca,<div><br></div><div>I just reproduced the exact same problem using only drbdadm. I also inspected the output of the -d option, and drbdadm does the exact same thing as Ganeti in this regard.</div><div>
<br></div><div>Thanks,</div><div>Thomas</div></div><div><div><div><br><br><div>On Tue, Apr 30, 2013 at 2:18 PM, Thomas Thrainer <span dir="ltr"><<a rel="nofollow" href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Luca,</div><div><br></div><div>I'll try the drbdadm method this as soon as I get to it. Right now I have no machines around which are already configured via drbdadm.</div>
<div><br></div><div>Note that I have included both logs (secondary and primary) in my first email.</div>
<div><br></div><div>I use DRBD 8.4.2 because 8.4.3 didn't compile against my kernel (3.2.27) and I didn't want to put much work into it. I this bug is known and fixed in 8.4.3, I'll might look into compiling against 3.2.27 again.</div>
<div><br></div><div>Cheers,</div><div><div>Thomas</div><div><div><br></div>-- <br><div dir="ltr"><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(213,15,37);padding-top:2px;margin-top:2px">Thomas Thrainer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(51,105,232);padding-top:2px;margin-top:2px"> Software Engineer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(0,153,57);padding-top:2px;margin-top:2px"> <a rel="nofollow" href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a> |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"> </span><br>
<div><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><br></span></div><div>
<span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="font-family:Arial,Verdana,sans-serif">Google Germany GmbH</span></div><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif"><div style="font-family:Arial,Verdana,sans-serif">
Dienerstr. 12</div><div style="font-family:Arial,Verdana,sans-serif">80331 München<br><br>Registergericht und -nummer: Hamburg, HRB 86891<br>Sitz der Gesellschaft: <font face="arial, sans-serif">Hamburg</font><br>Geschäftsführer: Graham Law, Katherine Stephens</div>
</div></span></div></div>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(213,15,37);padding-top:2px;margin-top:2px">Thomas Thrainer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(51,105,232);padding-top:2px;margin-top:2px"> Software Engineer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(0,153,57);padding-top:2px;margin-top:2px"> <a rel="nofollow" href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a> |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"> </span><br>
<div><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><br></span></div><div>
<span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="font-family:Arial,Verdana,sans-serif">Google Germany GmbH</span></div><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif"><div style="font-family:Arial,Verdana,sans-serif">
Dienerstr. 12</div><div style="font-family:Arial,Verdana,sans-serif">80331 München<br><br>Registergericht und -nummer: Hamburg, HRB 86891<br>Sitz der Gesellschaft: <font face="arial, sans-serif">Hamburg</font><br>Geschäftsführer: Graham Law, Katherine Stephens</div>
</div></span></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(213,15,37);padding-top:2px;margin-top:2px">Thomas Thrainer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(51,105,232);padding-top:2px;margin-top:2px"> Software Engineer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(0,153,57);padding-top:2px;margin-top:2px"> <a rel="nofollow" href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a> |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"> </span><br>
<div><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><br></span></div><div>
<span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="font-family:Arial,Verdana,sans-serif">Google Germany GmbH</span></div><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif"><div style="font-family:Arial,Verdana,sans-serif">
Dienerstr. 12</div><div style="font-family:Arial,Verdana,sans-serif">80331 München<br><br>Registergericht und -nummer: Hamburg, HRB 86891<br>Sitz der Gesellschaft: <font face="arial, sans-serif">Hamburg</font><br>Geschäftsführer: Graham Law, Katherine Stephens</div>
</div></span></div></div>
</div></div><br></div></div><div>_______________________________________________<br>drbd-user mailing list<br><a rel="nofollow" href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br><a rel="nofollow" href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br><br></div></div> </div> </div> </blockquote></div> </div></div><br>_______________________________________________<br>
drbd-user mailing list<br>
<a rel="nofollow" href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a rel="nofollow" href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br></blockquote></div><br></div></div><br>_______________________________________________<br>drbd-user mailing list<br><a rel="nofollow" href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a rel="nofollow" href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br><br></div> </div></div></div> </div> </blockquote></div> </div></div><br>_______________________________________________<br>
drbd-user mailing list<br>
<a rel="nofollow" href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a rel="nofollow" href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(213,15,37);padding-top:2px;margin-top:2px">Thomas Thrainer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(51,105,232);padding-top:2px;margin-top:2px"> Software Engineer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(0,153,57);padding-top:2px;margin-top:2px"> <a rel="nofollow" href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a> |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"> </span><br>
<div><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><br></span></div><div>
<span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="font-family:Arial,Verdana,sans-serif">Google Germany GmbH</span></div><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif"><div style="font-family:Arial,Verdana,sans-serif">
Dienerstr. 12</div><div style="font-family:Arial,Verdana,sans-serif">80331 München<br><br>Registergericht und -nummer: Hamburg, HRB 86891<br>Sitz der Gesellschaft: <font face="arial, sans-serif">Hamburg</font><br>Geschäftsführer: Graham Law, Katherine Stephens</div>
</div></span></div></div>
</div></div><br><br></div> </div></div></div> </div> </blockquote></div> </div></div><br>_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(213,15,37);padding-top:2px;margin-top:2px">Thomas Thrainer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(51,105,232);padding-top:2px;margin-top:2px"> Software Engineer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(0,153,57);padding-top:2px;margin-top:2px"> <a href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a> |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"> </span><br>
<div><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><br></span></div><div>
<span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="font-family:Arial,Verdana,sans-serif">Google Germany GmbH</span></div><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif"><div style="font-family:Arial,Verdana,sans-serif">
Dienerstr. 12</div><div style="font-family:Arial,Verdana,sans-serif">80331 München<br><br>Registergericht und -nummer: Hamburg, HRB 86891<br>Sitz der Gesellschaft: <font face="arial, sans-serif">Hamburg</font><br>Geschäftsführer: Graham Law, Katherine Stephens</div>
</div></span></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(213,15,37);padding-top:2px;margin-top:2px">Thomas Thrainer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(51,105,232);padding-top:2px;margin-top:2px"> Software Engineer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(0,153,57);padding-top:2px;margin-top:2px"> <a href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a> |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"> </span><br>
<div><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><br></span></div><div>
<span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="font-family:Arial,Verdana,sans-serif">Google Germany GmbH</span></div><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif"><div style="font-family:Arial,Verdana,sans-serif">
Dienerstr. 12</div><div style="font-family:Arial,Verdana,sans-serif">80331 München<br><br>Registergericht und -nummer: Hamburg, HRB 86891<br>Sitz der Gesellschaft: <font face="arial, sans-serif">Hamburg</font><br>Geschäftsführer: Graham Law, Katherine Stephens</div>
</div></span></div></div>
</div>