Category Archives: server

Running RHEL 6 on EFI BIOS-based system

Been ages since I updated the blog. Was really busy lately with lots of commitments and work-related task.

Just wanna share 1 thing to remember when you intend to run RHEL 6 on EFI-based BIOS. Few months ago I was tasked to setup RHEL 6 on an EFI-based system (IBM x-series 3850 M3 if not mistaken). Linux on IBM x series? should be a piece of cake rite? Just put the CD, setup the partition and voila. How wrong I was. It took me and a friend of mine (which is an expert in Linux with more than 10 years of experience, mind you) around 8 hours to got it right.

To make matters worse, we could not find any documentation regarding it on the Internet and even RH Knowledge Base and documentation didnt help either (at that time, dunno now). To shorten the matter, here are the simple guideline that I can give you. We are installing a RHEL 6 64bit OS BTW.

Linux boot the OS from the /boot partition. Problem is after the installation, it fails to even go to the bootloader menu. And installation is never completed without Anaconda raising a flag that it fails to install RHEL 6. Turn out to be the boot for EFI BIOS partition must be in vfat format in order for the system to detect it. So you have to mount /boot on EXT-based filesystem and /boot/EFI on vfat filesystem (Anaconda labels it as EFI boot partition if I am not mistaken). That should make the server able to boot the OS properly.

What really annoys me is that Anaconda didnt do the proper detection to flag us that our filesystem configuration is not workable. Or put something on the knowledge base about this issue. However if you do the installation using the default setup without any customization, it will actually detect you are installing on an EFI system and put the partition accordingly (Thats how we manage to solve this problem BTW).

Hope this can help some sysadmin from going back late at night to solve the same issue.

Red Hat (and KVM) are still RHEL-evant

I started to read with a bipartisan mindset about “Xen and Theory of RHEL-evance” posted in the Citrix community blog by Simon Crosby. What appears to be a great title at first seems to be mostly FUD on why KVM is doomed for failure especially in the enterprise marketplace and Red Hat will drown with it. It did not have enough facts, just FUD most of the time. I would to counter his so called “facts” here as its been a long time anyway since I last updated my blog.

Firstly Simon recommends that if you wanna go virtual with a Linux / RHEL mindset, that you should go with Oracle Enterprise Linux (OEL) because …

It is a superior, enterprise class version of RHEL, and typically more up to date than it, and OEL is guaranteed to be compatible with RHEL.

What did Simon means by superior, enterprise class version of RHEL? Typically more up to date? and being binary compatible at the same time? last time I check OEL is based on RHEL and using mostly CentOS patches. Can CentOS or OEL exists without RHEL at the first place? seems full of bull to me. And Oracle with its new shiny Solaris added to the mix, how do you think its gonna effect its Linux offerings?

Next, he also recommends to use SLES if you are non-Larry group …

Alternatively, if you’re wary of giving Larry more control than he already has over your environment, Novell SUSE Linux offers a superb enterprise Linux platform boasting more than 3,000 certified applications, fully supported on Xen and XenServer, with complete support for SAP and (via Mono) many Microsoft .Net apps.

As the world is moving to KVM, so do smart companies such as Novell. Its already a tech preview in SLES 11 and expected to be supported by SLES 11 SP1. So thanks Simon for recommending people to KVM-world. BTW welcome to the club Novell, great to have you onboard.

Availability of the KVM platform by Red Hat also seems to be another major issue. Simon noted that there’s no freely downloadable product for Red Hat KVM offering. Although I quite agree that RHEV is not freely available (for a reason that I will come to later), KVM support in RHEL is available since version 5.4 and for that matter OEL (from my source Oracle Linux team Australia seems to confirm KVM is in OEL) and CentOS have it on their distro. As I said earlier above, both were built from the source of RHEL and had the same packages, kernel version etc minus the RH branding. So go ahead download CentOS and run it as a Hypervisor for your VMs. If you feel the urge to get support, you know who to call.

The reason why Red Hat Enterprise Virtualization or RHEV is not available as open source is due to the current management console runs on .NET which requires Windows Server and other Microsoft dependencies. This is due to code being developed by Qumranet, a start-up that now part of Red Hat Inc. Red Hat is now heavily migrating all the tools into a Java-based application before releasing its source code (or say my Red Hat source). It will hopefully will be launched during RHEV version 3 soon. And I believe Red Hat due to its past history, where all its software being released fully bits by bits to GPL such as dogtag, 389ds, Teiid and JBoss.org initiative. BTW did Citrix release their Xen management tools as GPL?

Next up, history lesson.

Red Hat’s endeavors in virtualization started with profound endorsements of Xen, followed, when Novell SUSE Linux was the first vendor to ship Xen 3.0 in 2006 as a component of SLES 10, with an accusation that Novell was “irresponsible” and then by a completely unsubstantiated statement by Red Hat VP Alex Pinchev to ZDNet Australia, that “Xen is not stable yet, it’s not ready for the enterprise.”. Then, as RHEL 5 belatedly readied itself for delivery in late 2006, Red Hat proudly proclaimed that only it could deliver an enterprise class virtualization product based on Xen, together with a rich management infrastructure.

When Novell launched its Xen offering to the market with SLES 10, not long after that the Xen Community changes its default scheduler due to the rapid development of Xen at the time. Also Novell launched it offerings without any real management tools at that time (which later it includes in SP1).

Red Hat on the other hand launched with libvirt and virt-manager to protect its investment in the Virtualization marketplace. Libvirt is an API to hook to any virtualization technology underneath it OS so that if any new hot virtualization technology surfaced (which in this case KVM) it can then switch its current management tools (e.g. virsh and virt manager) to the new technology easily. More info on Libvirt here.

So IMHO Red Hat was “quite” right to say Xen is not ready when SLES 10 launch and put its money where its mouth is by developing libvirt and virt-manager which nearly all distro with any virtualization technologies bundles today.

Simon also seems to play down the relevance of KVM on his ending notes.

Essentially, Linus would have to agree to turn Linux into a great hypervisor, and historically he has maintained a balanced path that has not yielded to special interests.

Linux is meritocracy and only the best codes win. If the code is good enough, it will be in the kernel. KVM which written by Avi is a clean implementation of Virtualization on Linux just by enabling a module without rewriting the critical components of the kernel code. Why reinvent the wheel when some of the best and greatest mind in OS development is there in the mainstream kernel?

KVM architecture
KVM architecture which leverages mainstream Linux

KVM hypervisor needs a scheduler? thats already in Linux, KVM can use that. NUMA support? yes KVM can use a little bit of that etc etc. The slick implementation just by turning the module to turn your Linux as a hypervisor seems great to me anyway, more or so for Linus which approves it.

Do note that Xen is yet to be included into the mainstream kernel as of today.

The industry has three excellent type 1 hypervisors: Xen, Hyper-V and ESX. Does it really need a fourth, and a fourth that is incompatible with the others?

Hyper-V? Excellent? Really? I dont know if support for only 1 vcpu and flaky mouse integration for Linux considers as excellent.

Red Hat and the Linux community generally have failed to acknowledge the fundamental need in virtualization for a stable interface between the hypervisor and guests, with both backward and forward compatibility, for all time.

… and Citrix and Xen failed to acknowledge that we need to have only single standardized code base for everyone that is the mainstream Linux kernel. Why reinvent the wheel again?

Moreover the infrastructural “basics” of resource pooling, multi-tenancy, virtual switching, and virtual storage are quite simply missing from its offering.

While I do agree there is certain features not available in RHEV/KVM offerings, Xen do have its own deficiencies by itself. Kernel Samepage Merging (KSM) feature is still not yet available in the Xen Hypervisor which enables memory overcommitting. This will ensure you can run more RAM that you have as KVM will look for similarities in the memory footprint and merge them together. The only other virtualization technology that have this is VMWare last time I check. And NO, Memory Ballooning is not the same with KSM. I bet most customers wants KSM more than what you listed above. With KSM you can run more VMs on a smaller number of physical nodes for KVM than Xen.

kernel samepage merging
KSM in action

In other words, a large Linux distro, such as RHEL 6 with KVM, will have a substantially larger code base and attack surface than any type 1 hypervisor. Moreover it is likely to have to deal with every CERT update due to vulnerabilities in Linux. By contrast, a small embedded Xen hypervisor (together with a tiny embedded Linux kernel to run the management stack and drivers) can easily be delivered within a 16MB footprint.

Did Mr Simon knows that you can run a stateless hypervisor for RHEV and KVM under 128Mb using JeOS RHEL?. And yes we can also runs on memory card / USB flash drive etc a’la VMWare and Citrix thanks to the Fedora project livecd-to-iso script.

Finally, I have previously argued that the relevance of the OS-centric approach to next-gen infrastructure is questionable. The next-gen OS will not be a thing that runs a server – but a thing that runs apps across virtualized infrastructure.

Yeah but saying “OS-centric approach is questionable” and “Hyper-V is an excellent Hypervisor” in the same blog post is more questionable than your argument. Did Hyper-V is not based on Windows Server code base? Can we virtualize using Hyper-V on Windows Server 2008 R2?

Its business model, which requires that RHEL is a binary that is available only to paying customers, leaves it vulnerable to truly free, enterprise grade products.

Did you know you can run RHEV without paying for any RHEL? Just pick up how many sockets that you requires and runs RHEV-M and RHEV-H to virtualize your windows guest all without paying a single dollar for RHEL. Maybe this page can help you understand it better. You can email me if you need more info at our company website.

The end note is this, both Xen and KVM are wonderful piece of achievement by the community at large. Having both are a testament to the open source model itself and keeps each other honest and competitive product. By bashing KVM mindlessly and with FUD all over it doesn’t make Citrix Xen offering better and so do vice versa for Red Hat. And KVM do gain from strength to strength and now powering the big blue cloud and also The Planet. Not bad a piece of technology that just come around circa 2007.

Naturally for all Linux communities, distribution and vendors will favour the KVM approach due to its simplicity and tight integration with the kernel they knew and love. Red Hat, Fedora, SLES 11 SP1, OpenSUSE, Ubuntu, Debian, Mandriva etc had all integrate their offerings around KVM. But it is foolish to write off Xen, not when Oracle and Citrix are still behind it. Though I still thinks Citrix (and Simon) lost quite a few sleep due to KVM, RHEV and Red Hat lately.

Disclaimer : Warix Technologies is one of the partner for Red Hat Enterprise Virtualization in Malaysia and I am a Fedora Ambassador.

Red Hat Enterprise Virtualization (RHEV)

Disclaimer : Warix Technologies is one of the partner for Red Hat Enterprise Virtualization in Malaysia and I am a Fedora Ambassador. I have tested the late BETA of RHEV for a few days.

So Red Hat had already released their new and shiny Red Hat Enterprise Virtualization or RHEV in short, a few hours ago. This release, is one of the key milestone for RHEV after being officially announced in February 2009. So what this really means?

Virtualization is not something new, in fact its already nearly 50 years since IBM put their Virtualization solutions in their mainframe (and still is doing so on their System Z). But lately the surge of Virtualization hits new heights the last couple of years in the x86 market due to one factor, VMware.

Shortly after releasing their VMware workstation to the market in 1999 for the workstation and desktop market, VMware realized in order to be a big hit, it needs to play in the datacenter. So they start to focus their efforts to ensure their solution works reliably on the greatest of workloads and datacenters.

The Open Source Community picks up the fever soon after with the Xen Project in 2002. Xen the project and XenSource the company fuels the new technological goldmine and became for a while, the de facto standard. Citrix realized this and bought Xen for USD500 Million soon after.

Red Hat, Novell and other Open Source vendors jumps into the Xen fever and introduced it to the world circa 2006-2007 until now but hardly makes a dent into the VMware market either through lack of marketing, less features and I think most important of all IMHO – Manageability.

Around early 2007, enters KVM from Qumranet which turns Linux into a Hypervisor. But thats not all, they even build a user friendly web management interface that can manage big numbers of virtual machine, though mostly for Virtual Desktop Infrastructre (VDI) or desktop virtualization. Red Hat then acquired Qumranet and move the direction of the software to also manage the servers.

Enough of the history lesson, so basically RHEV have 4 components which are:-

  1. RHEV-M for Server (or Management Interface for Server) – This is basically the web-based GUI interface for RHEV solutions with ability to manage hundreds and thousands of server using its unique search-driven interface. This is the equivalent of Virtual Center of VMware.
  2. RHEV-M for Desktop (or Management Interface for Desktop) – Same with above but to manage your VDI Infrastructure. Also includes user portal for user to access their VDI. Release TBA.
  3. RHEV-H (RHEV Hypervisor Baremetal) – The baremetal Hypervisor based on the same code with RHEL 5.4 with less flexibility but easier installation for Non-Linux Admin / User. It can also be installed on USB disk, memory cards on CD.
  4. Red Hat Enterprise Linux (RHEL) 5.4 with KVM support – The vanilla RHEL that we know and love (or CentOS for some of you) with KVM and RHEV management support. This basically means you can hook-up your RHEL into the RHEV-M to manage your servers.

So what’s great about all this RHEV stuff?

  1. KVM – Yes that is single biggest thing in KVM that will be the game changer. Unlike Xen, the application compatibility betwen Linux in Baremetal and Linux in Virtual environment is gone as the 2 share the same code base. So whatever runs in baremetal, will run under KVM without any modification. This will ensure compatibility between binaries and ensure less headache to software vendors to create version specialized for VM environments.
  2. Manageability – Virt Manager to manage big numbers of VMs? With all due respect, NO. Enter RHEV-M. The one massive advantage that VMware had over other Open Source Virtualization vendors will be no more soonish. RHEV-M enables administrators to manage their Virtual infrastructures without knowing the command line that Linux is famous for. Its point and clickfest all over on their favourite Internet Explorer’s .. yes IE and that one will be talked about later :p
  3. Open Source – When talking about Open Source, there’s 2 great things. 1 is the cost factor and no 2 is the features factor. The offerings from Redhat is cheap compared to its competitors (USD499 for standard and USD7xx for premium per socket) and the features will keep on coming (libvirt, SRIOV, Libguestfs etc) when you subscribe to RHEV due to active community around KVM and its companion.
  4. Memory and Storage Overcommit – Thanks to QCOW or Thin Provisioning and KSM or memory page paging, its now possible to allocate more memory (RAM) to your server up to 150% and same applies to your storage (though maybe not 150%).
  5. Paravirt Drivers – RHEL 4.8 (and above) and RHEL 5.3 (and above) automatically accelerated with VirtIO Paravirt Drivers that will boost the performance of the VM to near native. For Windows, the paravirt drivers is WHQL certified and will be delivered through Windows Update. Cool rite? Thats what you get when you in bed with the devil!

What is NOT so great about it?

  1. M$ - The biggest issue about RHEV (and most controversial) is you have to run the management web interface on Windows Server. Yes its not a typo. And its also needs to have (take a breath) MS SQL, Windows Presentation Foundation, Active Directory, Windows POWERSHELL and also ONLY supports Internet Exploder.. opps I mean Explorer 6,7 and 8. And NO you cannot run it on Mono or Wine or Crossover for all the stuff above. But on the other hand, the potential customer might even be OK with it as they already have their Windows Infrastructure in their organization.
  2. Scripting – You can now do scripting to your RHEV, but only supports POWERSHELL!! no bash for you for now, though the VMware guys might be OK with it as VMware also supports powershell.
  3. I cant think of any more major roadblock, So just a placeholder if I remember any.

All in all, RHEV is a solid initial product, thanks to Redhat, Qumranet, Open Source Model and all the communities. This is what I call the power of participation. Although they are certain things that doesnt appeal for purist and Open Source community, I think Redhat have explored all the possibilities and thinks this the right way to go for now.

The new version (which will be RHEV 3, the current is RHEV 2.1) which targeted by middle of next year will runs on JBoss and Hibernate (which means you can hook it to almost all database) which should make everyone happy except VMware off course.