<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We would like to properly configure DRBD9 using RAM disk bdev’s as an underlying devices on both ends for testing purposes.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So far we have seen quite unsatisfactory results using Linux kernel’s BRD module for RAM disks and the following DRBD resource config:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">resource no_cas_test {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> device /dev/drbd0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> on lab_1 {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> disk /dev/ram0; #cache dev<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> meta-disk internal;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> address XXXX<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> on lab_2 {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> disk /dev/ram0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> meta-disk internal;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> address XXXX<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">}<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">These are HW machines running RHEL and the raw performance we get from RAM disks is over 16GiB/s (over 4400k IOPS) using fio:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"># fio --filename=/dev/ram0 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=24 --name=4kiops_randwrite --group_reporting --size=4G<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas">...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas">fio-3.19<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas">Starting 24 processes<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas">Jobs: 16 (f=14): [w(1),_(1),w(5),_(1),w(1),_(1),f(2),_(5),w(7)][85.7%][w=15.3GiB/s][w=4023k IOPS][eta 00m:01s]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">4kiops_randwrite: (groupid=0, jobs=24): err= 0: pid=3976: Fri Dec 3 15:53:43 2021<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:Consolas"> write: IOPS=4402k, BW=16.8GiB/s (18.0GB/s)(96.0GiB/5717msec); 0 zone resets<o:p></o:p></span></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">However, as soon as we start a single, primary DRBD node on lab_1, we only get about 1300MiB/s (350k IOPS).<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"># fio --filename=/dev/drbd0 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=24 --name=4kiops_randwrite --group_reporting --size=4G<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">fio-3.19<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Starting 24 processes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Jobs: 24 (f=24): [w(24)][98.6%][w=1379MiB/s][w=353k IOPS][eta 00m:01s]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">4kiops_randwrite: (groupid=0, jobs=24): err= 0: pid=4274: Fri Dec 3 15:59:22 2021<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> write: IOPS=351k, BW=1371MiB/s (1438MB/s)(96.0GiB/71704msec); 0 zone resets<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Now if we use physical NVMe drives, there is no difference in performance – both direct hardware (/dev/nvmeXXXXX) test and DRBD test yield maximum drive write performance:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Non-DRBD test:<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"># fio --filename=/dev/nvme1n1p1 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=12 --name=4kiops_randwrite --group_reporting --size=4G<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">fio-3.19<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Starting 12 processes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Jobs: 1 (f=1): [_(11),w(1)][98.7%][w=607MiB/s][w=155k IOPS][eta 00m:01s]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">4kiops_randwrite: (groupid=0, jobs=12): err= 0: pid=5312: Fri Dec 3 12:34:05 2021<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:Consolas"> write: IOPS=165k, BW=646MiB/s (677MB/s)(48.0GiB/76092msec); 0 zone resets<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> slat (nsec): min=1419, max=310283, avg=3730.55, stdev=2879.91<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> clat (usec): min=5, max=23109, avg=4231.60, stdev=2497.87<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> lat (usec): min=9, max=23113, avg=4235.39, stdev=2497.50<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> clat percentiles (usec):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> | 1.00th=[ 86], 5.00th=[ 545], 10.00th=[ 1074], 20.00th=[ 1991],<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> | 30.00th=[ 2638], 40.00th=[ 3425], 50.00th=[ 4080], 60.00th=[ 4817],<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> | 70.00th=[ 5538], 80.00th=[ 6063], 90.00th=[ 7177], 95.00th=[ 8848],<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> | 99.00th=[11469], 99.50th=[12256], 99.90th=[14091], 99.95th=[14877],<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> | 99.99th=[16909]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> bw ( KiB/s): min=351792, max=1961536, per=100.00%, avg=725184.65, stdev=26445.60, samples=1653<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> iops : min=87948, max=490384, avg=181296.12, stdev=6611.40, samples=1653<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> </span>
<span lang="PL" style="font-size:8.0pt;font-family:Consolas">lat (usec) : 10=0.05%, 20=0.23%, 50=0.44%, 100=0.38%, 250=1.11%<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas"> lat (usec) : 500=2.36%, 750=2.48%, 1000=2.23%<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas"> lat (msec) : 2=10.84%, 4=28.80%, 10=48.27%, 20=2.80%, 50=0.01%<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas">
</span><span style="font-size:8.0pt;font-family:Consolas">cpu : usr=3.19%, sys=7.10%, ctx=8562282, majf=0, minf=718<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> issued rwts: total=0,12582912,0,0 short=0,0,0,0 dropped=0,0,0,0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> latency : target=0, window=0, percentile=100.00%, depth=64<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Run status group 0 (all jobs):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> WRITE: bw=646MiB/s (677MB/s), 646MiB/s-646MiB/s (677MB/s-677MB/s), io=48.0GiB (51.5GB), run=76092-76092msec<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Disk stats (read/write):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> nvme1n1: ios=205/12567629, merge=0/0, ticks=197/53105077, in_queue=46882592, util=100.00%<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">DRBD test:<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"># fio --filename=/dev/drbd0 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=12 --name=4kiops_randwrite --group_reporting --size=4G<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">fio-3.19<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Starting 12 processes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Jobs: 1 (f=1): [_(6),w(1),_(5)][95.1%][w=636MiB/s][w=163k IOPS][eta 00m:04s]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">4kiops_randwrite: (groupid=0, jobs=12): err= 0: pid=5104: Fri Dec 3 12:28:03 2021<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:Consolas"> write: IOPS=162k, BW=634MiB/s (665MB/s)(48.0GiB/77474msec); 0 zone resets<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> slat (nsec): min=1951, max=198869k, avg=49104.65, stdev=79888.62<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> clat (usec): min=7, max=207233, avg=4469.84, stdev=2174.11<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> lat (usec): min=13, max=207304, avg=4519.00, stdev=2178.99<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> clat percentiles (usec):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> | 1.00th=[ 498], 5.00th=[ 1500], 10.00th=[ 1860], 20.00th=[ 2376],<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> | 30.00th=[ 3064], 40.00th=[ 3818], 50.00th=[ 4555], 60.00th=[ 5211],<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> | 70.00th=[ 5669], 80.00th=[ 6128], 90.00th=[ 6849], 95.00th=[ 7832],<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> | 99.00th=[10159], 99.50th=[11076], 99.90th=[13435], 99.95th=[15270],<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> | 99.99th=[27395]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> bw ( KiB/s): min=411961, max=1893680, per=100.00%, avg=677991.74, stdev=20345.92, samples=1765<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> iops : min=102987, max=473420, avg=169496.99, stdev=5086.49, samples=1765<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> </span>
<span lang="PL" style="font-size:8.0pt;font-family:Consolas">lat (usec) : 10=0.01%, 20=0.02%, 50=0.04%, 100=0.06%, 250=0.19%<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas"> lat (usec) : 500=0.76%, 750=1.45%, 1000=0.74%<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas"> lat (msec) : 2=9.48%, 4=29.65%, 10=56.43%, 20=1.15%, 50=0.03%<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas"> lat (msec) : 250=0.01%<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas"> cpu : usr=1.90%, sys=62.47%, ctx=811802, majf=0, minf=1305<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PL" style="font-size:8.0pt;font-family:Consolas">
</span><span style="font-size:8.0pt;font-family:Consolas">IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> issued rwts: total=0,12582912,0,0 short=0,0,0,0 dropped=0,0,0,0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> latency : target=0, window=0, percentile=100.00%, depth=64<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Run status group 0 (all jobs):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> WRITE: bw=634MiB/s (665MB/s), 634MiB/s-634MiB/s (665MB/s-665MB/s), io=48.0GiB (51.5GB), run=77474-77474msec<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">Disk stats (read/write):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> drbd0: ios=0/12560419, merge=0/0, ticks=0/47859223, in_queue=47859223, util=100.00%, aggrios=0/12582912, aggrmerge=0/0, aggrticks=0/47319227, aggrin_queue=41223461, aggrutil=100.00%<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas"> nvme1n1: ios=0/12582912, merge=0/0, ticks=0/47319227, in_queue=41223461, util=100.00%<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have seen even worse performance on the VM’s running DRBD9 on RAM disks. Is that behavior expected?
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best regards,<o:p></o:p></p>
<p class="MsoNormal">Krzysztof Majzerowicz-Jaszcz<o:p></o:p></p>
</div>
<p>
<hr/>
<strong style="line-height: 11.25pt;">
<span style="font-size: 9pt; color: #595959;">
<span style="font-family: 'Arial Narrow', sans-serif;">Intel Technology Poland sp. z o.o.<br>
</span>
</span>
</strong>
<span style="color:#595959; font-family: 'Arial Narrow', sans-serif; font-size: 9pt; line-height: 11.25pt;">
ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk Północ | VII Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP 957-07-52-316 | Kapitał zakładowy 200.000 PLN.
</span>
</p>
<p>
<span style="font-size:8.0pt; font-family:'Arial Narrow',sans-serif; mso-fareast-font-family: 'Times New Roman';mso-bidi-font-family:Arial; color:#595959; mso-ansi-language:EN-US; mso-fareast-language:EN-US; mso-bidi-language: PL-PL">
Ta wiadomość wraz z załącznikami jest przeznaczona dla określonego adresata i może zawierać informacje poufne. W razie przypadkowego otrzymania tej wiadomości, prosimy o powiadomienie nadawcy oraz trwałe jej usunięcie; jakiekolwiek przeglądanie lub rozpowszechnianie jest zabronione.<br>
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
</span>
</p>
<p class="MsoNormal"><o:p></o:p></p>
</body>
</html>