<div dir="ltr"><div dir="ltr">You're welcome.</div><div dir="ltr"><br></div><div dir="ltr">With LVM you will have separated resource for each VM in the end.<br>The only thing is snapshots, unfortunately they are not working for LVM-storage.<br>And there is no way for use thinlvm in shared mode.<br><br>PS: Ow, I thought that I've sent it to drbd-user list. OK, I'll send there copy too :)<br><div><div><div dir="ltr" class="gmail_signature"><div dir="ltr">- kvaps<br></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Oct 4, 2018 at 2:47 PM Yannis Milios <<a href="mailto:yannis.milios@gmail.com">yannis.milios@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Andrey,<div><br></div><div>Thanks for sharing your setup, which is actually interesting. However, in my setup I prefer to have separate DRBD resources per VM, hence the need to have something like LINSTOR (or drbdmanage before that) to automate resource/volume creation and management. I'm using it for both QEMU and LXC and so far it's working great. </div><div>As a backing storage I'm using LVM Thin (and ZFS Thin sometimes), which means that I can have instant snapshots, per VM/LXC with the ability to rollback at any time instantly. In this setup there's no need to use an iSCSI target as you do in your setup, as LINSTOR plugin for Proxmox take care for everything.</div><div><br></div><div>Regards,</div><div>Yannis</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, 4 Oct 2018 at 13:37, kvaps <<a href="mailto:kvapss@gmail.com" target="_blank">kvapss@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi, I've using DRBD9 on three nodes Proxmox-cluster without LINSTOR, it is working fine.</div><div><br></div><div>You probably should prepare one big DRBD-device replicated to three nodes.</div><div><br></div><div> # cat /etc/drbd.d/tgt1.res<br> resource tgt1 {<br> meta-disk internal;<br> device /dev/drbd100;<br> protocol C;<br> net { <br> after-sb-0pri discard-zero-changes;<br> after-sb-1pri discard-secondary;<br> after-sb-2pri disconnect;<br> }<br> on pve1 {<br> address <a href="http://192.168.2.11:7000" target="_blank">192.168.2.11:7000</a>;<br> disk /dev/disk/by-partuuid/95e7eabb-436e-4585-94ea-961ceac936f7;<br> node-id 0;<br> }<br> on pve2 {<br> address <a href="http://192.168.2.12:7000" target="_blank">192.168.2.12:7000</a>;<br> disk /dev/disk/by-partuuid/aa7490c0-fe1a-4b1f-ba3f-0ddee07dfee3;<br> node-id 1;<br> }<br> on pve3 {<br> address <a href="http://192.168.2.13:7000" target="_blank">192.168.2.13:7000</a>;<br> disk /dev/disk/by-partuuid/847b9713-8c00-48a1-8dff-f84c328b9da2;<br> node-id 2;<br> }<br> connection-mesh {<br> hosts pve1 pve2 pve3;<br> }<br> }<br></div><div><br></div><div>Then you can create LXC container, which will use this block device as it's rootfs.</div><div><br></div><div> mkfs -t ext4 /dev/drbd100<br> wget <a href="http://download.proxmox.com/images/system/ubuntu-16.04-standard_16.04-1_amd64.tar.gz" target="_blank">http://download.proxmox.com/images/system/ubuntu-16.04-standard_16.04-1_amd64.tar.gz</a> -P /var/lib/vz/template/cache/<br> pct create 101 local:vztmpl/ubuntu-16.04-standard_16.04-1_amd64.tar.gz \<br> --hostname=tgt1 \<br> --net0=name=eth0,bridge=vmbr0,gw=192.168.1.1,ip=<a href="http://192.168.1.11/24" target="_blank">192.168.1.11/24</a> \<br> --rootfs=volume=/dev/drbd100,shared=1<br></div><div> pct start 101</div><div><br></div><div>Login into container and create one big file there.</div><div><br></div><div> pct exec 101 bash<br> mkdir -p /data<br> fallocate -l 740G /data/target1.img<br></div><div><br></div><div><div>Install istgt, and configure iSCSI-export for this file.</div><div><br></div><div> # cat /etc/istgt/istgt.conf<br> [Global]<br> Comment "Global section"<br> NodeBase "iqn.2018-07.org.example.tgt1"<br> PidFile /var/run/istgt.pid<br> AuthFile /etc/istgt/auth.conf<br> MediaDirectory /var/istgt<br> LogFacility "local7"<br> Timeout 30<br> NopInInterval 20<br> DiscoveryAuthMethod Auto<br> MaxSessions 16<br> MaxConnections 4<br> MaxR2T 32<br> MaxOutstandingR2T 16<br> DefaultTime2Wait 2<br> DefaultTime2Retain 60<br> FirstBurstLength 262144<br> MaxBurstLength 1048576<br> MaxRecvDataSegmentLength 262144<br> InitialR2T Yes<br> ImmediateData Yes<br> DataPDUInOrder Yes<br> DataSequenceInOrder Yes<br> ErrorRecoveryLevel 0<br> [UnitControl]<br> Comment "Internal Logical Unit Controller"<br> AuthMethod CHAP Mutual<br> AuthGroup AuthGroup10000<br> Portal UC1 <a href="http://127.0.0.1:3261" target="_blank">127.0.0.1:3261</a><br> Netmask 127.0.0.1<br> [PortalGroup1]<br> Comment "SINGLE PORT TEST"<br> Portal DA1 <a href="http://192.168.1.11:3260" target="_blank">192.168.1.11:3260</a><br> [InitiatorGroup1]<br> Comment "Initiator Group1"<br> InitiatorName "ALL"<br> Netmask <a href="http://192.168.1.0/24" target="_blank">192.168.1.0/24</a><br> [LogicalUnit1]<br> Comment "Hard Disk Sample"<br> TargetName disk1<br> TargetAlias "Data Disk1"<br> Mapping PortalGroup1 InitiatorGroup1<br> AuthMethod Auto<br> AuthGroup AuthGroup1<br> UseDigest Auto<br> UnitType Disk<br> LUN0 Storage /data/target1.img Auto<br></div></div><div><br></div><div>On the Proxmox go to the storage interface, and connect this export to all three nodes. (remove checkmark from "Use LUN Directly")<br></div><div> <a href="https://hsto.org/webt/uw/j3/pu/uwj3pusr-nf9bc7neisd5x-fcsg.png" target="_blank">https://hsto.org/webt/uw/j3/pu/uwj3pusr-nf9bc7neisd5x-fcsg.png</a></div><div><br></div><div></div><div>After that you can create one shared LVM storage on top of iSCSI device via Proxmox interface. (maek "shared" checkmark)<br></div><div> <a href="https://hsto.org/webt/j1/ob/mw/j1obmwcwhz-e6krjix72pmiz118.png" target="_blank">https://hsto.org/webt/j1/ob/mw/j1obmwcwhz-e6krjix72pmiz118.png</a><br></div><div><br></div><div>Also don't forget configure HA for your container:</div><div><br></div><div> ha-manager groupadd tgt1 --nodes pve1,pve2,pve3 --nofailback=1 --restricted=1<br> ha-manager add ct:101 --group=tgt1 --max_relocate=3 --max_restart=3<br></div><div><br></div><div>After those steps you will have shared storage for all three nodes, so you can create and live-migrate your VMs without any problems.</div><div>And migrate iSCSI-container with short downtime.</div><div><br></div><div>Cheers<br></div><div><div><div><div><div><div><div dir="ltr" class="gmail-m_7779014479370803106m_-6057131300438041269gmail_signature"><div dir="ltr">- kvaps<br></div></div></div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Oct 4, 2018 at 12:27 PM Yannis Milios <<a href="mailto:yannis.milios@gmail.com" target="_blank">yannis.milios@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">You can, but your life will be miserable without LINSTOR managing the resources (hence the existence of it in the first place) ... :)</div><br><div class="gmail_quote"><div dir="ltr">On Wed, 3 Oct 2018 at 13:29, M. Jahanzeb Khan <<a href="mailto:mjahanzebkhan@yahoo.com" target="_blank">mjahanzebkhan@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:Helvetica Neue,Helvetica,Arial,sans-serif;font-size:13px"><div style="font-family:Helvetica Neue,Helvetica,Arial,sans-serif;font-size:13px">Hello,<br><br>I would like to know that is it possible to use drbd 9 without using Linstor on top of LVM ?<br>I have a 3 nodes server and I was using drbdmanage before. But now I just want to use drbd 9 without any additional tools.<br><br><br>Best regards,<br>Jaz</div></div><div id="gmail-m_7779014479370803106m_-6057131300438041269m_-2872065238668172421m_-7213948685694810749DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br>
<table style="border-top:1px solid rgb(211,212,222)">
<tbody><tr>
<td style="width:55px;padding-top:13px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" style="width: 46px; height: 29px;" width="46" height="29"></a></td>
<td style="width:470px;padding-top:12px;color:rgb(65,66,78);font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">Virus-free. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link" style="color:rgb(68,83,234)" target="_blank">www.avast.com</a>
</td>
</tr>
</tbody></table><a href="#m_7779014479370803106_m_-6057131300438041269_m_-2872065238668172421_m_-7213948685694810749_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div></div>_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</blockquote></div>
_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></div></div></div>