<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.gmail-il
{mso-style-name:gmail-il;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">For anyone else who is considering using VDO over DRBD. This is what we discovered about getting it to work.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Sweet Tea Dorminy <sweettea@redhat.com> <br>
<b>Sent:</b> Tuesday, July 6, 2021 1:50 PM<br>
<b>To:</b> vdo-devel <vdo-devel@redhat.com><br>
<b>Subject:</b> Getting snapshotting working with VDO: a recent experience.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Recently I had the pleasure of helping someone figure out how to use VDO with their snapshotting solution, and figured I'd send out a summary, with their permission, to the list in case it helps anyone else.<o:p></o:p></p>
<div>
<p class="MsoNormal">Setup: Eric was using DRBD for data replication, and VDO on top to provide data reduction, similarly to
<a href="https://linbit.com/blog/albireo-virtual-data-optimizer-vdo-on-drbd/">Linbit's article</a>. The stack looked like this:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Physical -> LVM -> DRBD -> VDO -> FileSystem<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Specifically:
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/dev/sda<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> /dev/vg/lv
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> /dev/drbd0<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> /dev/vdo0
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> /my-filesystem (xfs)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Problem: Eric wanted to add snapshotting to the mix, taking a snapshot and verifying that it matched the filesystem. Without VDO in the stack, this is a simple matter of taking a snapshot
of /dev/vg/lv and mounting the snapshot. With VDO, though, how to get from /dev/vg/lv-snap to a verifiable filesystem?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- 'vdo create --device=/dev/vg/lv-snap' doesn't work -- it formats a new VDO, and complains if it finds an already existing vdo.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- 'vdo import' doesn't work -- it complains there's already a VDO with the same UUID on the system.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- just mounting /dev/vg/lv-snap doesn't work, as it is a vdo, not a filesystem.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Solving: In order to start a VDO from storage containing an already formatted VDO, 'vdo import' is necessary, but as said, it reported a UUID collision. After some searching, we
realized passing "--uuid=''" (the empty string) to vdo import would import the snapshot VDO and also change the UUID, so it wouldn't collide with the original VDO.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">For instance:<br>
<span style="font-family:"Courier New"">[sweettea@localhost ~]$ sudo <span class="gmail-il">
vdo</span> create --name vdo0 --device /dev/fedora_localhost-live/vms<br>
Creating <span class="gmail-il">VDO</span> vdo0<br>
Logical blocks defaulted to 6802574 blocks.<br>
The <span class="gmail-il">VDO</span> volume can address 26 GB in 13 data slabs, each 2 GB.<br>
It can grow to address at most 16 TB of physical storage in 8192 slabs.<br>
If a larger maximum size might be needed, use bigger slabs.<br>
Starting <span class="gmail-il">VDO</span> vdo0<br>
Starting compression on <span class="gmail-il">VDO</span> vdo0<br>
<span class="gmail-il">VDO</span> instance 0 volume is ready at /dev/mapper/vdo0<br>
[sweettea@localhost ~]$ sudo lvcreate --size 2G --snapshot --name vms_snap /dev/fedora_localhost-live/vms<br>
Logical volume "vms_snap" created.<br>
[sweettea@localhost ~]$ sudo <span class="gmail-il">vdo</span> <span class="gmail-il">
import</span> --name vdo0-snap --device /dev/fedora_localhost-live/vms_snap<br>
<span class="gmail-il">Importing</span> <span class="gmail-il">VDO</span> vdo0-snap<br>
<span class="gmail-il">vdo</span>: ERROR - UUID df61bc46-13dc-4091-bdec-4896c744888c already exists in
<span class="gmail-il">VDO</span> volume(s) stored on /dev/disk/by-id/dm-uuid-LVM-Gexh1cit2vwmcvIf2AAvullg3mWvrnql0SluovtpLXaYGoyPw84zRD4NPmGO3Nu4<br>
[sweettea@localhost ~]$ sudo <span class="gmail-il">vdo</span> <span class="gmail-il">
import</span> --name vdo0-snap --device /dev/fedora_localhost-live/vms_snap --uuid<br>
usage: <span class="gmail-il">vdo</span> <span class="gmail-il">import</span> [-h] -n <volume> --device <devicepath> [--activate {disabled,enabled}] [--blockMapCacheSize <megabytes>] [--blockMapPeriod <period>] [--compression {disabled,enabled}]<br>
[--deduplication {disabled,enabled}] [--emulate512 {disabled,enabled}] [--maxDiscardSize <megabytes>] [--uuid <uuid>] [--vdoAckThreads <threadCount>]<br>
[--vdoBioRotationInterval <ioCount>] [--vdoBioThreads <threadCount>] [--vdoCpuThreads <threadCount>] [--vdoHashZoneThreads <threadCount>]<br>
[--vdoLogicalThreads <threadCount>] [--vdoLogLevel {critical,error,warning,notice,info,debug}] [--vdoPhysicalThreads <threadCount>]<br>
[--writePolicy {async,async-unsafe,sync,auto}] [-f <file>] [--logfile <pathname>] [--verbose]<br>
<span class="gmail-il">vdo</span> <span class="gmail-il">import</span>: error: argument --uuid: expected one argument<br>
[sweettea@localhost ~]$ sudo <span class="gmail-il">vdo</span> <span class="gmail-il">
import</span> --name vdo0-snap --device /dev/fedora_localhost-live/vms_snap --uuid ''<br>
<span class="gmail-il">Importing</span> <span class="gmail-il">VDO</span> vdo0-snap<br>
Starting <span class="gmail-il">VDO</span> vdo0-snap<br>
Starting compression on <span class="gmail-il">VDO</span> vdo0-snap<br>
<span class="gmail-il">VDO</span> instance 1 volume is ready at /dev/mapper/vdo0-snap<br>
<br>
</span>(The documentation on --uuid was somewhat confusing, but passing an empty string worked.)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">However, mounting the resulting snapshot still didn't work.
<br>
'journalctl -K' showed these logs from a different attempt:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714073.665039] kvdo3:dmsetup: underlying device, REQ_FLUSH: supported, REQ_FUA: not supported</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714073.665041] kvdo3:dmsetup: Using write policy async automatically.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714073.665042] kvdo3:dmsetup: loading device 'snap_vdo0'</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714073.665055] kvdo3:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714073.724597] kvdo3:dmsetup: starting device 'snap_vdo0'</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714073.724607] kvdo3:journalQ: Device was dirty, rebuilding reference counts</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714074.025715] kvdo3:journalQ: Finished reading recovery journal</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714074.032016] kvdo3:journalQ: Highest-numbered recovery journal block has sequence number 4989105, and the highest-numbered usable block is 4989105</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714074.337654] kvdo3:physQ0: Replaying entries into slab journals for zone 0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714074.536446] kvdo3:physQ0: Recreating missing journal entries for zone 0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714074.536503] kvdo3:journalQ: Replayed 6365157 journal entries into slab journals</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714074.536504] kvdo3:journalQ: Synthesized 0 missing journal entries</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714074.538423] kvdo3:journalQ: Saving recovery progress</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714074.803255] kvdo3:logQ0: Replaying 2545622 recovery entries into block map</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714077.211236] kvdo3:journalQ: Flushing block map changes</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714077.231360] kvdo3:journalQ: Rebuild complete.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714077.296258] kvdo3:journalQ: Entering recovery mode</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714077.296460] kvdo3:dmsetup: Setting UDS index target state to online</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714077.296473] kvdo3:dmsetup: device 'snap_vdo0' started</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714077.296474] kvdo3:dmsetup: resuming device 'snap_vdo0'</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714077.296476] kvdo3:dmsetup: device 'snap_vdo0' resumed</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714077.296493] uds: kvdo3:dedupeQ: loading or rebuilding index: dev=/dev/disk/by-id/dm-uuid-LVM-CKGi2ErfgK2rWEYpTJg0OhkXrkf7bRCAIzSDMIJdu5kf0SZFdqgtOySFCarMb1Cl offset=4096 size=2781704192</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714077.339599] uds: kvdo3:dedupeQ: Using 6 indexing zones for concurrency.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714077.448849] kvdo3:packerQ: compression is enabled</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714078.142975] uds: kvdo3:dedupeQ: loaded index from chapter 0 through chapter 85</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714078.227367] uds: kvdo3:dedupeQ: Replaying volume from chapter 3929 through chapter 4952</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714078.233636] uds: kvdo3:dedupeQ: unexpected index page map update, jumping from 85 to 3929</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714082.392773] kvdo3:journalQ: Exiting recovery mode</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<b><span style="font-family:Consolas;color:red">[3714134.313962] XFS (dm-8): Filesystem has duplicate UUID be2e762b-6dea-48c4-83ad-08ece1cac43b - can't mount</span></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714156.566380] uds: kvdo3:dedupeQ: replay changed index page map update from 85 to 4951</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714320.750730] kvdo3:dmsetup: suspending device 'snap_vdo0'</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714320.751156] uds: dmsetup: beginning save (vcn 4951)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714321.135314] uds: dmsetup: finished save (vcn 4951)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714321.135322] kvdo3:dmsetup: device 'snap_vdo0' suspended</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714321.135360] kvdo3:dmsetup: stopping device 'snap_vdo0'</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">[3714321.201264] kvdo3:dmsetup: device 'snap_vdo0' stopped</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Based on the red line, the remaining problem was a XFS uuid collision, much like the VDO uuid collision on the first attempt to vdo import. Searching revealed
<a href="https://www.miljan.org/main/2009/11/16/lvm-snapshots-and-xfs/">this link</a> which suggested either xfs_admin -U generate or mount -o nouuid to change or ignore the uuid of the XFS filesystem. Using these, Eric was able to use the setup fully: taking
a snapshot, starting VDO on the snapshot, and mounting the filesystem stored thereon.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Hope this helps someone else in the future!<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Sweet Tea<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this
email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility
for any loss or damage arising from the use of this email or attachments.
</body>
</html>