<div dir="ltr">Hi Thomas,<div><br></div><div style>From Ganeti point of view it makes perfectly sense to use drbdadm directly (I'm not familiar using it this way) and you are right drbdadm is "just" a wrapper to drbdsetup. Anyway you should give a try at "drbdadm -d --allow-two-primaries=yes r0". Please note the -d switch that just do a dry-run and print out the relative drbdsetup invocation.</div>
<div style><br></div><div style>But look at your log file:<br></div><div style><div style="font-family:arial,sans-serif;font-size:13px">Apr 26 11:19:42 primary kernel: [181721.669870] block drbd1: peer( Secondary -> Primary ) </div>
<div style="font-family:arial,sans-serif;font-size:13px">Apr 26 11:19:42 primary kernel: [181722.057848] d-con resource1: sock was shut down by peer</div><div style="font-family:arial,sans-serif;font-size:13px">Apr 26 11:19:42 primary kernel: [181722.057872] d-con resource1: peer( Primary -> Unknown ) conn( Connected -> BrokenPipe ) pdsk( UpToDate -> DUnknown ) </div>
<div style="font-family:arial,sans-serif;font-size:13px">Apr 26 11:19:42 primary kernel: [181722.057881] d-con resource1: short read (expected size 16)</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div>
<div style="font-family:arial,sans-serif;font-size:13px">Soon after resource1 changed his state from secondary to primary on the remote node the network socket was closed by the remote node itself.</div><div style="font-family:arial,sans-serif;font-size:13px">
What are the remote side logs saying?</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">I'm not aware of any bug on 8.4.2 related to dual primary but I can be wrong. Altough 8.4.3 is out.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Cheers,</div><div style="font-family:arial,sans-serif;font-size:13px">Luca</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Apr 29, 2013 at 9:42 AM, Thomas Thrainer <span dir="ltr"><<a href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Luca,<div><br></div><div>we (Ganeti) use drbdsetup directly because it's the more "programmatic" way of configuring DRBD. We don't want to manage the configuration file, but just reconfigure machines again as they join a cluster. You could think of Ganeti as managing the DRBD configuration in a different way.</div>
<div><br></div><div>Anyway, drbdadm is just a wrapper around drbdsetup. In particular, "<span style="font-family:arial,sans-serif;font-size:13px">drbdadm --allow-two-primaries=yes r0" just calls "</span><span style="font-family:arial,sans-serif;font-size:13px">drbdsetup net-options ipv4:<local_ip>:11001 ipv4:<remote_ip>:11001 --allow-two-</span><span style="font-family:arial,sans-serif;font-size:13px">primaries=yes" (according to </span><a href="http://git.drbd.org/gitweb.cgi?p=drbd-8.4.git;a=blob;f=user/drbdadm_main.c;h=8179625f7bef7172c07974dd63bf76ddb10b0d60;hb=HEAD#l1668" target="_blank">http://git.drbd.org/gitweb.cgi?p=drbd-8.4.git;a=blob;f=user/drbdadm_main.c;h=8179625f7bef7172c07974dd63bf76ddb10b0d60;hb=HEAD#l1668</a>). Can adding "--protocol=C" really make a difference? Especially if --allow-two-primaries only works with protocol C anyways?</div>
<div>Additionally, the actual problem occurs as soon as I issue a "drbdsetup primary 0" (which is what "drbdadm primary r0" calls as well).</div><div><br></div><div>As I stated, I tried to call the above command on one and on both sides simultaneously, with the same outcome.</div>
<div><br></div><div>So, am I hitting a DRBD bug here? Or do you have other ideas of what I could do wrong?</div><div><br></div><div>Cheers,</div><div>Thomas</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra">
<br><br><div class="gmail_quote">On Fri, Apr 26, 2013 at 5:04 PM, Luca Fornasari <span dir="ltr"><<a href="mailto:luca.fornasari@gmail.com" target="_blank">luca.fornasari@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Thomas,<div><br></div><div>In line reply below<br><div class="gmail_extra"><div><br>On Fri, Apr 26, 2013 at 4:14 PM, Thomas Thrainer <span dir="ltr"><<a href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a>></span> wrote:<br>
</div><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Luca,<div><br></div><div>(CC'd drbd-user, I guess that might be helpful for others as well)</div>
</div></blockquote><div><br></div></div><div>Just reply to the list; I'm subscribed ;)</div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>We're not using drbdadm but drbdsetup directly.<br></div><div><br></div>
<div>I tried `drbdsetup net-options ipv4:<local_ip>:11001 ipv4:<remote_ip>:11001 --protocol C --allow-two-</div><div>primaries=yes` (i.e. I stripped the repeated options), but the result is still the same.</div>
</div></blockquote><div><br></div></div><div>I' not 100% sure but I think that repeating ipv4:<local_ip>:local_port ipv4:<remote_ip>:remote_port restart the connection; during off-load time that happens fast enough while during high-load fails.</div>
<div>Just try to use "drbdadm --allow-two-primaries=yes r0" on one node only.</div><div>Do you have a good reason to use drbdsetup directly?</div><div><br></div><div>Cheers,</div><div>
Luca</div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">
<div>Note however, that the problem occurs only every now and then, and primarily when there is load on the disk(s).</div><div><br></div><div>BTW, I actually do set two disks to dual-primary mode at the same time (using different connections/resources tough), and one disk normally works while the other fails (is't not deterministic which of disk fails).</div>
<div><br></div><div>Cheers,</div><div>Thomas</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Apr 26, 2013 at 3:58 PM, Luca Fornasari <span dir="ltr"><<a href="mailto:luca.fornasari@gmail.com" target="_blank">luca.fornasari@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Thomas,<div><br></div><div>Just execute the following on one node only:</div><div><br></div><div>drbdadm net-options --protocol=C --allow-two-primaries r0</div>
<div><br></div><div>
I guess that the command you are issuing just try to restart an already running resource.</div><div><br></div><div>Cheers,</div><div>Luca</div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>
On Fri, Apr 26, 2013 at 2:27 PM, Thomas Thrainer <span dir="ltr"><<a href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div>
<div dir="ltr">Hi,<div><br></div><div>I've encountered a problem with DRBD 8.4.2 when I try to enable --allow-two-primaries on the fly and immediately promoting the secondary to primary afterwards.</div><div>The problem doesn't occur always, and it seems like it is more likely to happen when there is more load on the device.</div>
<div><br></div><div>The exact command sequence is as follows:</div><div><br></div><div>Executed on primary and secondary node simultaneously (but also happens if only executed on secondary):</div><div><br></div><div>drbdsetup net-options ipv4:<loc_ip>:11001 ipv4:<rem_ip>:11001 --protocol C --after-sb-0pri discard-zero-changes --after-sb-1pri consensus --allow-two-primaries=yes --cram-hmac-alg md5 --shared-secret <secret></div>
<div>drbdsetup primary 1</div><div><br></div><div>BTW, the only options which differs in regard to the previously issued drbdsetup connect command is --allow-two-primaries. The rest (protocol, secret, etc.) are just repeated.</div>
<div><br></div><div>The outcome is that both nodes end up in the StandAlone state.</div><div><br></div><div>Their respective kernel log messages are:</div><div><br></div><div>(Old) primary:</div>
<div><div>Apr 26 11:19:42 primary kernel: [181721.646750] block drbd0: peer( Secondary -> Primary ) </div><div>Apr 26 11:19:42 primary kernel: [181721.669870] block drbd1: peer( Secondary -> Primary ) </div><div>
Apr 26 11:19:42 primary kernel: [181722.057848] d-con resource1: sock was shut down by peer</div><div>Apr 26 11:19:42 primary kernel: [181722.057872] d-con resource1: peer( Primary -> Unknown ) conn( Connected -> BrokenPipe ) pdsk( UpToDate -> DUnknown ) </div>
<div>Apr 26 11:19:42 primary kernel: [181722.057881] d-con resource1: short read (expected size 16)</div><div>Apr 26 11:19:42 primary kernel: [181722.057914] block drbd1: new current UUID DEEF411AB544C5D3:041691050BDB6491:FA1A2A8EC7D3D7CE:FA192A8EC7D3D7CF</div>
<div>Apr 26 11:19:42 primary kernel: [181722.057964] d-con resource1: asender terminated</div><div>Apr 26 11:19:42 primary kernel: [181722.057977] d-con resource1: Terminating asender thread</div><div>Apr 26 11:19:42 primary kernel: [181722.058485] d-con resource1: Connection closed</div>
<div>Apr 26 11:19:42 primary kernel: [181722.067019] d-con resource1: conn( BrokenPipe -> Unconnected ) </div><div>Apr 26 11:19:42 primary kernel: [181722.067027] d-con resource1: receiver terminated</div><div>Apr 26 11:19:42 primary kernel: [181722.067032] d-con resource1: Restarting receiver thread</div>
<div>Apr 26 11:19:42 primary kernel: [181722.067036] d-con resource1: receiver (re)started</div><div>Apr 26 11:19:42 primary kernel: [181722.067045] d-con resource1: conn( Unconnected -> WFConnection ) </div><div>Apr 26 11:19:43 primary kernel: [181722.558370] d-con resource1: Handshake successful: Agreed network protocol version 101</div>
<div>Apr 26 11:19:43 primary kernel: [181722.558702] d-con resource1: Peer authenticated using 16 bytes HMAC</div><div>Apr 26 11:19:43 primary kernel: [181722.558747] d-con resource1: conn( WFConnection -> WFReportParams ) </div>
<div>Apr 26 11:19:43 primary kernel: [181722.558754] d-con resource1: Starting asender thread (from drbd_r_resource [2039])</div><div>Apr 26 11:19:43 primary kernel: [181722.560436] block drbd1: drbd_sync_handshake:</div>
<div>Apr 26 11:19:43 primary kernel: [181722.560445] block drbd1: self DEEF411AB544C5D3:041691050BDB6491:FA1A2A8EC7D3D7CE:FA192A8EC7D3D7CF bits:3072 flags:0</div><div>Apr 26 11:19:43 primary kernel: [181722.560454] block drbd1: peer 9CE29D13EEB7B4B3:041691050BDB6491:FA1A2A8EC7D3D7CF:FA192A8EC7D3D7CF bits:0 flags:0</div>
<div>Apr 26 11:19:43 primary kernel: [181722.560466] block drbd1: uuid_compare()=100 by rule 90</div><div>Apr 26 11:19:43 primary kernel: [181722.560474] block drbd1: helper command: /bin/true initial-split-brain minor-1</div>
<div>Apr 26 11:19:43 primary kernel: [181722.565127] d-con resource1: conn( WFReportParams -> NetworkFailure ) </div><div>Apr 26 11:19:43 primary kernel: [181722.565134] d-con resource1: asender terminated</div><div>Apr 26 11:19:43 primary kernel: [181722.565138] d-con resource1: Terminating asender thread</div>
<div>Apr 26 11:19:43 primary kernel: [181722.570459] block drbd1: helper command: /bin/true initial-split-brain minor-1 exit code 0 (0x0)</div><div>Apr 26 11:19:43 primary kernel: [181722.570488] block drbd1: helper command: /bin/true split-brain minor-1</div>
<div>Apr 26 11:19:43 primary kernel: [181722.583047] block drbd1: helper command: /bin/true split-brain minor-1 exit code 0 (0x0)</div><div>Apr 26 11:19:43 primary kernel: [181722.583073] d-con resource1: conn( NetworkFailure -> Disconnecting ) </div>
<div>Apr 26 11:19:43 primary kernel: [181722.583143] d-con resource1: Connection closed</div><div>Apr 26 11:19:43 primary kernel: [181722.586237] d-con resource1: conn( Disconnecting -> StandAlone ) </div><div>Apr 26 11:19:43 primary kernel: [181722.586245] d-con resource1: receiver terminated</div>
<div>Apr 26 11:19:43 primary kernel: [181722.586249] d-con resource1: Terminating receiver thread</div><div>Apr 26 11:19:46 primary kernel: [181726.054479] br974: port 2(vif126.0) entering forwarding state</div><div>Apr 26 11:19:46 primary kernel: [181726.058824] br974: port 2(vif126.0) entering disabled state</div>
<div><br></div><div>(Old) secondary:</div><div><div>Apr 26 11:19:42 secondary kernel: [1809212.315376] block drbd0: role( Secondary -> Primary ) </div><div>Apr 26 11:19:42 secondary kernel: [1809212.338517] block drbd1: role( Secondary -> Primary ) </div>
<div>Apr 26 11:19:42 secondary kernel: [1809212.726247] d-con resource1: peer( Primary -> Unknown ) conn( Connected -> ProtocolError ) pdsk( UpToDate -> DUnknown ) </div><div>Apr 26 11:19:42 secondary kernel: [1809212.726278] block drbd1: new current UUID 9CE29D13EEB7B4B3:041691050BDB6491:FA1A2A8EC7D3D7CF:FA192A8EC7D3D7CF</div>
<div>Apr 26 11:19:42 secondary kernel: [1809212.726310] d-con resource1: asender terminated</div><div>Apr 26 11:19:42 secondary kernel: [1809212.726340] d-con resource1: Terminating asender thread</div><div>Apr 26 11:19:42 secondary kernel: [1809212.726719] d-con resource1: Connection closed</div>
<div>Apr 26 11:19:42 secondary kernel: [1809212.726749] d-con resource1: conn( ProtocolError -> Unconnected ) </div><div>Apr 26 11:19:42 secondary kernel: [1809212.726755] d-con resource1: receiver terminated</div><div>
Apr 26 11:19:42 secondary kernel: [1809212.726759] d-con resource1: Restarting receiver thread</div><div>Apr 26 11:19:42 secondary kernel: [1809212.726763] d-con resource1: receiver (re)started</div><div>Apr 26 11:19:42 secondary kernel: [1809212.726771] d-con resource1: conn( Unconnected -> WFConnection ) </div>
<div>Apr 26 11:19:43 secondary kernel: [1809213.226864] d-con resource1: Handshake successful: Agreed network protocol version 101</div><div>Apr 26 11:19:43 secondary kernel: [1809213.227199] d-con resource1: Peer authenticated using 16 bytes HMAC</div>
<div>Apr 26 11:19:43 secondary kernel: [1809213.227238] d-con resource1: conn( WFConnection -> WFReportParams ) </div><div>Apr 26 11:19:43 secondary kernel: [1809213.227245] d-con resource1: Starting asender thread (from drbd_r_resource [20607])</div>
<div>Apr 26 11:19:43 secondary kernel: [1809213.231289] block drbd1: drbd_sync_handshake:</div><div>Apr 26 11:19:43 secondary kernel: [1809213.231297] block drbd1: self 9CE29D13EEB7B4B3:041691050BDB6491:FA1A2A8EC7D3D7CF:FA192A8EC7D3D7CF bits:0 flags:0</div>
<div>Apr 26 11:19:43 secondary kernel: [1809213.231306] block drbd1: peer DEEF411AB544C5D3:041691050BDB6491:FA1A2A8EC7D3D7CE:FA192A8EC7D3D7CF bits:3072 flags:0</div><div>Apr 26 11:19:43 secondary kernel: [1809213.231315] block drbd1: uuid_compare()=100 by rule 90</div>
<div>Apr 26 11:19:43 secondary kernel: [1809213.231322] block drbd1: helper command: /bin/true initial-split-brain minor-1</div><div>Apr 26 11:19:43 secondary kernel: [1809213.232460] block drbd1: helper command: /bin/true initial-split-brain minor-1 exit code 0 (0x0)</div>
<div>Apr 26 11:19:43 secondary kernel: [1809213.232494] block drbd1: helper command: /bin/true split-brain minor-1</div><div>Apr 26 11:19:43 secondary kernel: [1809213.233512] block drbd1: helper command: /bin/true split-brain minor-1 exit code 0 (0x0)</div>
<div>Apr 26 11:19:43 secondary kernel: [1809213.233539] d-con resource1: conn( WFReportParams -> Disconnecting ) </div><div>Apr 26 11:19:43 secondary kernel: [1809213.233574] d-con resource1: asender terminated</div><div>
Apr 26 11:19:43 secondary kernel: [1809213.233579] d-con resource1: Terminating asender thread</div><div>Apr 26 11:19:43 secondary kernel: [1809213.233631] d-con resource1: Connection closed</div><div>Apr 26 11:19:43 secondary kernel: [1809213.233662] d-con resource1: conn( Disconnecting -> StandAlone ) </div>
<div>Apr 26 11:19:43 secondary kernel: [1809213.233667] d-con resource1: receiver terminated</div><div>Apr 26 11:19:43 secondary kernel: [1809213.233672] d-con resource1: Terminating receiver thread</div><div><br></div><div>
<br></div><div>What am I doing wrong? Is there a requirement to wait for a sync/propagation of properties/random amount of time before promoting the secondary to primary? Is this a bug?</div><div><br></div><div>
Thanks,</div><div>Thomas</div></div></div><span><font color="#888888"><div><div><br></div>-- <br><div dir="ltr"><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(213,15,37);padding-top:2px;margin-top:2px">Thomas Thrainer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(51,105,232);padding-top:2px;margin-top:2px"> Software Engineer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(0,153,57);padding-top:2px;margin-top:2px"> <a href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a> |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"> </span><br>
<div><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><br></span></div><div>
<span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="font-family:Arial,Verdana,sans-serif">Google Germany GmbH</span></div><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif"><div style="font-family:Arial,Verdana,sans-serif">
Dienerstr. 12</div><div style="font-family:Arial,Verdana,sans-serif">80331 München<br><br>Registergericht und -nummer: Hamburg, HRB 86891<br>Sitz der Gesellschaft: <font face="arial, sans-serif">Hamburg</font><br>Geschäftsführer: Graham Law, Katherine Stephens</div>
</div></span></div></div>
</div></font></span></div>
<br></div></div>_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(213,15,37);padding-top:2px;margin-top:2px">Thomas Thrainer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(51,105,232);padding-top:2px;margin-top:2px"> Software Engineer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(0,153,57);padding-top:2px;margin-top:2px"> <a href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a> |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"> </span><br>
<div><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><br></span></div><div>
<span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="font-family:Arial,Verdana,sans-serif">Google Germany GmbH</span></div><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif"><div style="font-family:Arial,Verdana,sans-serif">
Dienerstr. 12</div><div style="font-family:Arial,Verdana,sans-serif">80331 München<br><br>Registergericht und -nummer: Hamburg, HRB 86891<br>Sitz der Gesellschaft: <font face="arial, sans-serif">Hamburg</font><br>Geschäftsführer: Graham Law, Katherine Stephens</div>
</div></span></div></div>
</div>
</div></div></blockquote></div></div></div><br></div></div></div>
<br>_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(213,15,37);padding-top:2px;margin-top:2px">Thomas Thrainer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(51,105,232);padding-top:2px;margin-top:2px"> Software Engineer |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(0,153,57);padding-top:2px;margin-top:2px"> <a href="mailto:thomasth@google.com" target="_blank">thomasth@google.com</a> |</span><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"> </span><br>
<div><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><br></span></div><div>
<span style="color:rgb(85,85,85);font-family:sans-serif;font-size:small;line-height:1.5em;border-width:2px 0px 0px;border-style:solid;border-color:rgb(238,178,17);padding-top:2px;margin-top:2px"><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="font-family:Arial,Verdana,sans-serif">Google Germany GmbH</span></div><div style="font-size:13px;line-height:normal;color:rgb(34,34,34);font-family:arial,sans-serif"><div style="font-family:Arial,Verdana,sans-serif">
Dienerstr. 12</div><div style="font-family:Arial,Verdana,sans-serif">80331 München<br><br>Registergericht und -nummer: Hamburg, HRB 86891<br>Sitz der Gesellschaft: <font face="arial, sans-serif">Hamburg</font><br>Geschäftsführer: Graham Law, Katherine Stephens</div>
</div></span></div></div>
</div>
</div></div></blockquote></div><br></div>