Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi again, well, O_DIRECT is a Linux specific and only used in very special cases, thus mostly not used by default or at least deactivatable. To be fair, this is not a DRBD problem, but Linux', as it allows user space processes to bypass parts of the kernel for tuning -- you may call it fast path, but actually it is a bypass including side effects. In the past there have even been several discussions in the Linux kernel community on dropping O_DIRECT support (or replacing it by something more sane), but it was kept for what I believe don't-break-the-userland compatibility reasons. I am quite confident this will only apply to few of your containers. So the actual task is to identify them, which might already been done due to the issues you encountered. Also I suppose that LXC allows to run a process in some kind of super context, just like Linux VServers/Linux Secure Contexts and OpenVZ does, so a process on the host can see all processes from all the contexts at once. So running lsof for diagnostics in this super context should give you a list of all files currently open. Use the +fG option to add a column to the listing that shows all the flags used to open the files -- O_DIRECT is 0x8 and is ORed bitwise to the other flags. You might want to use +fg instead, which decodes the flags, but the decoded flags are abbreviated and I do not know from memory what abbreviation is used for O_DIRECT. O_DIRECT might also be safe in combination with O_SYNC, as I suppose O_SYNC to prevent in-flight changes of buffers by blocking writes until data is processed, but that question should be asked to and answered by someone with more Linux kernel (source code) expertise. Best regards, // Veit -------- Ursprüngliche Nachricht -------- Von: Christoph Lechleitner <christoph.lechleitner at iteg.at> Gesendet: 28. Dezember 2017 21:14:37 MEZ An: drbd-user <drbd-user at lists.linbit.com> CC: Veit Wahlich <cru.lists at zodia.de>, Wolfgang Glas <wolfgang.glas at iteg.at> Betreff: Re: [DRBD-user] Semantics of oos value, verification abortion On 2017-12-28 13:32, Veit Wahlich wrote: > Hi Christoph, > > I do not have experience with the precise functioning of LXC disk storage, but I assume that every operation that could cause oos applies to every application running inside the LXC containers, too. > > A common cause, that I suspect here, is opening a file (or block device) using O_DIRECT. This flag is used to reduce I/O latency and especially bypass the page cache, but it also allows buffers to be modified in-flight while they are processed by e.g. DRBD. So not only DRBD is affected by this, but also software RAID such as mdraid, dmraid or lvmraid, and I bet even block caching such as bcache. Are you serious? Can someone from linbit please comment on this? This would basically mean that DRBD is useless whenever an application opens files with O_DIRECT!? How could a fast path to user space render the replication of the underlying block device useless? > In most cases O_DIRECT is used by applications such as some DBMS to avoid caching by the kernel, as they implement their own cache or do not want the kernel to sacrifice memory on page caching as the data written will not be used again. > > So my recommendation is to check your logs/monitoring if the oos has only occurred repeatedly on certain containers, and then inspect the applications' configuration running inside for the use of O_DIRECT (which can usually be disabled). > If it has been occurring on all your containers, I would suspect your LXC configuration itself as the cause, such as an overlay filesystem or container image. Checking 1000s of applications in 100s of containers is NOT an option. Regards, Christoph