Open Source Software


Part 1 of the series Yum Repositories for Corporate Deployments.

Before we talk about how to set up a new yum repository, we need to discuss why, so that we can set up the right kind of repository. Before we can discuss why to set up a new repository, we need to know what exactly can be set up.

Terminology

Repository

A yum repository is a collection of rpm packages explained by a series of metadata databases. These databases are described in a XML file called “repomd.xml”, and all of this metadata is created automatically by a package appropriately named “createrepo.”

yum

yum is a client (meaning it reaches out to another server) tool to discover software availability from repositories and, if instructed, install that software onto the rpm-compatible GNU/Linux computer. yum uses the rpm database and format to determine dependencies, then handles those dependencies automatically.  For example, if a piece of software requires “python” and python is not installed, yum will find the python package and install it at its latest version, from any of the repositories it has been told of from its repo files.

repo

A repo file is a description to the yum client about where to find a repository, and what to do with the data found therein.  A repo file can describe multiple repositories, that are generally related.  Each repository definition starts with the repository name, and will include a path to the repository (the exported directory on the server which contains the “repodata” metadata directory that describes the repository), the user-friendly name of the repository, and some information about security (gpg signing and protection status).

 Background – Base Repositories

A default Red Hat Enterprise Linux (RHEL, even though Red Hat doesn’t like it) or CentOS or Scientific Linux installation ships with one or several repo files pointing to several different repositories.  Understanding these base repositories helps us decide what we want to set up inside our corporate network.  There are at least 2 default repositories in that configuration, “base”, “updates”.  CentOS and Scientific Linux also ship “extras”, “plus”, and CentOS ships “contrib”.

base

The “base” repository is the installation copy.  If you’re building a “base” repository in your corporate environment, you simply copy the entire DVD image to the repository location and leave it.  “base” is used to build an initial system to a known state.

updates

Updates is the repository where rpm updates for security, etc. are actually updated.  If you’re building an “updates” repository in your corporate network, you need to sync it from another repository synced to the {RHEL,CentOS,Scientific} original repository.  If you use the correct rsync command, you *only* need to rsync the repository (more in part 2).

extras, plus, and contrib

The best explanations of these repositories for CentOS come from the CentOS team themselves here: http://wiki.centos.org/AdditionalResources/Repositories

Repository basics

At this point, the repository names are just names.  Only “base” and “updates” have any real meaning – “base” is for the CD install, and “updates” is for security/other updates to “base”.  Any other repository is named so that the client system administrator can understand what might be installed from that repository.

The CentOS Team has some great information, which I’m lifting from http://wiki.centos.org/AdditionalResources/Repositories :

  • Use of hard-coded version and architecture: ‘baseurl=http: //ftp.belnet.be/packages/dries.ulyssis.org/redhat/el4/en/i386/dries/RPMS’ This hard codes both for ‘$releasever’ and ‘$basearch’. Compare this, to the more proper: ‘baseurl=http: //apt.sw.be/redhat/el$releasever/en/$basearch/dag’. The ‘hard coded’ approach limits it to only be ‘correct’ for CentOS 4 on an i386 platform.
  • Mixing Fedora repositories with CentOS oriented repositories: Look for ‘name=Fedora’, vs. ‘name=CentOS.(whatever)’. Fedora repositories are not likely to be compatible with CentOS. Repositories for other Enterprise Linux distros derived from the same upstream sources are more likely to be compatible, but should still be used with care.

The same rules will apply to building a corporate yum repository.  For building a corporate repository, I would also add: It’s easier to create more repositories, rather than trying to merge existing ones.  If you are syncing parts of repositories, they should be separate repositories, synced as a single unit.  Yum’s configuration is able to handle many repo files with many repository definitions, as long as those repositories don’t install the same files with conflicting versions.

Series

Continue to Part 2: Setting up a corporate yum repository mirror for bandwidth and staged update management

Return to the series header: Yum Repositories for Corporate Deployments

I was tasked with a documentation project regarding use of yum repositories inside a customer’s environment and realized that I was having a hard time finding exactly the information I needed to build the full test environment required to test the documentation.  What will follow is a series of posts covering information I had to research as background.

Topics

  1. Background, Terminology and repository basics
  2. Setting up a corporate yum repository mirror for bandwidth and staged update management
  3. Setting up a private repository for additional software

Notes

As a quick note up front, based on what I’ve learned, I can’t find a good reason to include the software from a private repository in the update management repository – just build a new repository.

PXE Booting – there are instructions here which will break a PXE Boot environment. Our lab doesn’t PXE or Net boot, so we haven’t tested against them, but a colleague assures me it won’t work.

Bibliography

The following posts were used to build this series. None of them was 100% what I needed, and no source had everything I needed in a single location, but they were all useful, and in no order at all:
http://www.howtoforge.com/creating_a_local_yum_repository_centos
http://wiki.centos.org/HowTos/CreateLocalMirror
http://kenfallon.com/how-to-mirror-rhn-behind-your-firewall/
https://docs.fedoraproject.org/en-US/Fedora/14/html/Software_Management_Guide/ch08s04.html
http://www.outsidaz.org/blog/2012/02/19/using-reposync-to-provide-local-yumrpm-mirrors/

EDIT

There’s a new version of this post at https://www.totalnetsolutions.net/2017/09/13/kubuntu-17-04-on-dell-e5470-for-extreme-battery-life/, since it’s 4 years newer, you should head there instead.

/EDIT

It’s been a long time since I’ve updated my Linux buildout on this site.

I’ve recently upgraded to a Lenovo T430 (from a T500, and Dell before that). This laptop has the following hardware:

Jump to main sections with these links:
CPU Configuration
Network Configuration
Video/Bumblebee Configuration
Encryption / Security configuration
Battery saving configuration
Custom kernel .config
This is the first time in years I haven’t had a built-in 3G Modem for internet access, but with wifi tethering to my phone, I don’t think it’ll be an issue. Out of the box, everything I cared about worked. I didn’t test the fingerprint reader or the Nvidia graphics with the default install, and pretty quickly customized the system, but if you’re not into customization, rest assured, the Ubuntu team did a fantastic job on the setup.

CPU and Battery


As stated in this post from 2007, I’m a huge fan of extreme battery life. I’m still using cpufreqd, laptop-mode-tools, and checking their configuration with powertop to make sure I’m doing everything I can. I also custom-compile my kjernel, which I’ll discuss more below.

Kubuntu 12.04 defaults to the “ondemand” cpufreq driver, which is great for power savings, except that it does its speed modulation in preference of performance. That is, when the lowest speed of the processor (1.2GHz in my case) isn’t enough, ondemand immediately jumps the CPU speed to the fastest available (2.9GHz in my case). Then, when the fastest is more than required, ondemand steps off slowly. This is perfect for video and gaming applications, and most people. I, however, greatly prefer the “conservative” driver, which works in the opposite manner: when an application needs more power, the conservative driver steps up 1 CPU speed level at a time, until the appropriate CPU speed is reached. Then, when the utilization drops off, conservative immediately drops all the way to the slowest speed, to step back up again if needed.
Using cpufreqd allows me to control this even more granularily, while not getting in the way of the kernel modules. My configuration uses “ondemand” when I’m plugged in, and “conservative” when I’m on battery power. The small delay in performance is worth the added minutes in battery life, especially since most of my on-battery time is very low-demand applications.
Since sometime in 2011, Ubuntu, however, has not shipped a working cpufreqd daemon – it’s apparently broken in the upstream Debian as well, and is well documented in this Launchpad bug. So I downloaded the cpufreqd-dev source package, the patch, and rebuilt cpufreqd myself. Now that it’s working, I can use the attached updated cpufreqd.conf configuration.


Network

Both the ethernet and wireless adapters work right on first install with Kubuntu 12.10.  The wireless uses the iwlwifi driver, and connects to my router at 104mbps.  I’m still using the script from this post to get my MTU set to 9000, rather than 1500, for jumbo frames support when in the main office network. This has a major effect on network speed for large transfers, but most networks don’t support it.h

The Wireless adapter needed no changes, and NetworkManager handles it beautifully, even when tethering to my phone.

Sound


I haven’t had any sound issues, but a few users have reported problems when using the docking station. According to this post on ThinkPad Forums, the solution is to simply edit /etc/modprobe.d/alsa-base.conf and add:
options snd-hda-intel model=thinkpad
This causes no issues on my system.

Video


This laptop ships with two video cards, an Intel (which uses the kernel i915 driver) low-power adapter, and an Nvideo high-performance, high-power adapter. In Windows, you can click an application to switch between the two for all applications. The Nvidia adapter uses about 10W more power than the Intel card, which means using the Nvidia adapter alone halves the system’s battery life.
By default, the Ubuntu kernel enables a feature called “vgaswitcheroo” which is well documented on the Ubuntu help site. I had a hard time getting it to work with my custom kernel, though, even though it was enabled. KDE and lightdm just didn’t want to swtich the laptop panel over to the Nvidia card. This *may* have something to do with my BIOS settings, which I currently can’t change due to office IT restrictions.
There is a new project called Bumblebee, which allows the user to use the Intel card, and only turn the Nvidia on for some applications. This gives the best of both worlds for power and battery savings, but is a work-in-progress, and not all applications run under Bumblebee. I’m using the Primus additions. Installation of Bumblebee is documented here, and Primus installation is documented here. I didn’t have to make any changes to get these installs working with simple applications.


VMware Workstation 9

VMware Workstation 9, however, offered some interesting challenges with Bumblebee. Out of the box, Vmware Workstation 9 installed (even with my custom kernel), and ran great, but would always give a warning that 3D acceleration was not available, which I expected while using the Intel card. However, Bumblebee has some limitations which mean it can’t run VMware workstation by default. I wrote a script to handle this, which I wrote up last night. I’m using a lot of work of others, so follow the links on that page to cmillersp’s post on the VMware Community Forums.


Encryption and Security

During installation, I chose the option to use an encrypted LVM volume. This uses DM-Crypt to encrypt the full HDD, so that it has to be unlocked at boot time. The Kubuntu installer seems to forget this fact, so it also asks you to set up ecryptfs private home directories, which is NOT neccessary for a single-user laptop, since the whole OS is already encrypted. The only oddity with dm-crypt is that sometimes the splash screen prompt to unlock the computer doesn’t show. If I just wait for disk activity to disappear, and have a blank screen, I can just type the passphrase, and it’ll still unlock successfully.

I don’t have the fingerprint reader set up, but if I do, I’ll update this post.


Battery and Power Savings

First, I use the configuration in the CPU Configuration section above for cpufreqd. Then I use laptop-mode-tools to set other configuration settings. I’ve attached all the files I have modified here, and it’s fairly power-saving aggressive. The only thing I should do, but don’t, is to disable the bluetooth adapter when I’m on battery, since it uses 1W just for the adapter. However, I have bluetooth headphones and a bluetooth mouse, which is why I have the bluetooth adapter in the laptop to begin with, so disabling it removes some critical functionality. I DO have it set to autosuspend, which is a little annoying when I go back to the mouse after 5 seconds of inactivity, but the annoyance is worth the savings, especially when I’m writing a long post like this.
laptop-mode.conf
All of these go in /etc/laptop-mode/conf.d :
auto-hibernate.conf
cpufreq.conf
ethernet.conf
intel-hda-powersave.conf
video-out.conf

With these settings, I was able to write this whole post, with the bluetooth mouse connected, running at an average of 13.2W. In 3.4 hours (I was doing other tasks, including feeding and changing my napping baby), I used 41% of my 93.6Wh battery. If I were to take this on a plane, I’d kill the wireless and bluetooth for probably another 2W savings, but I’d do that by hand.


Kernel Configuration

I have been building a custom kernel for my laptop for about 6 years now. The default Ubuntu image uses “generic-x86_64” for the processor family, but all of my laptops are “Core2 / Newer Xeon”. Just making that single change to my kernel results in about 0.5W-1W less power consumption, due to the increased efficiency gained by the kernel knowing about the new processor registers and commands that aren’t available to older processors. This greater CPU efficiency also means lower temperatures, and therefore lower fan speed.

My kernel configuration is attached here. Build it by following the instructions at the Ubuntu Help site

sudo apt-get install linux-kernel-devel fakeroot kernel-wedge build-essential
sudo apt-get install linux-source
sudo usermod -a -G src YOUR_USERNAME

Now log out and back in, so that you’re a member of the “src” group.

cd /usr/src
sudo chown -R $USER:src .
tar -jxf ./linux-source-3.5.0/linux-source-3.5.0.tar.bz2
ln -s linux-source-3.5.0 linux
cd linux
wget http://www.totalnetsolutions.net/wp-content/uploads/2012/12/rob-config-20121204c.txt
mv rob-config-20121204c.txt .config
make oldconfig
make menuconfig

Make any changes you want in here, then exit and save

fakeroot make-kpkg --initrd --append-to-version=.20121204c kernel_image kernel_headers

You’ll get 2 DEB files in /usr/src that you can then install and boot to. the “append-to-version” I use as a dating system for my kernels. “20121204c” means the 4th kernel attempt on December 4th, 2012, the day I recieved this laptop.

All Modified Files


Laptop-mode-tools config:
laptop-mode.conf
All of these go in /etc/laptop-mode/conf.d :
auto-hibernate.conf
cpufreq.conf
ethernet.conf
intel-hda-powersave.conf
video-out.conf
Other configs:
custom kernel .config.
cpufreqd.conf
Thanks.

I have a new Lenovo T430 with Nvidia/Intel hybrid graphics. The Intel card is for power saving, the Nvidia card for performance. The easiest way to handle this setup is to just choose the correct card on bootup, but this is inconvenient. Bumblebee provides a way to use the Intel card most of the time, but use the Nvidia for high-performance tasks such as games, and ONLY for those apps. Once it’s set up, it works pretty well, except for a few apps that require additional tweaking, such as Steam, or Wine.

One of the apps that requires tweaking is VMware Workstation (I’m running WKS 9). cmillersp provided a great write-up on the VMware Communities, which is what I set up on my system. VMware Workstation runs great – I can load OpenGL 3D apps in VMs and everything runs fantastic.

Except that I run VMs all the time, and using the Nvidia card all the time kills the performance benefit of the Intel card twice: once because I’m using the Nvidia card, and once again because BOTH GPUs are running at the same time. So I wanted a way to dynamically choose which card to run VMware under, based on whether I was on AC power or battery power.

The result is the attached script below, which I’ll be submitting to the Bumblebee wiki / project, as well as the VMware forums. This is version 1, which works as follows:

  1. Must be installed by “sudo ~/bin/vmware –install” – it will make the cd /usr/lib/vmware/bin; mv vmware-vmx vmware-vmx.real and then do the script creation at /usr/lib/vmware/bin/vmware-vmx, so that it’s a single portable script.
  2. Takes several options to force using the Nvidia card or not
  3. If no force option is applied, automatically determines AC adapter state or battery charge/discharge state
  4. Based on the above, decides whether to launch /usr/bin/vmware normally, or via the instructions from cmillersp

I’ve done some additional work to make it run via either “primusrun” or “optirun”. Optirun is much slower, but at least functional with fewer installs. I hope this is useful to someone else!

Edit: 2012-12-17 – v1.1 added gksu auto-detection
Edit: 2014-01-08 – Moved to github repo: github.com/docsmooth/vmware-bumblebee

I was troubleshooting a problem with some other vendor’s software tonight on a Red Hat Enterprise Linux 5.3 system.  We were able to reproduce the problem in the lab,  which was a huge boost to production, and insight, but we hit a wall when we got this error:

GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-42.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /opt/vendor/redacted/process...(no debugging symbols found)...done.
Attaching to program: /opt/vendor/redacted/process, process 16492
ptrace: Operation not permitted.

The weird thing is that we were running gdb as root, and it was 2.6.18. In the latest Ubuntu versions, a security hardening option has been added to the kernel to limit gdb (profiling, particularly, which gdb requires) to only being run on child processes. Since this was Red Hat Enterprise Linux 5.3, it didn’t have this option.

Well, it turns out that explaining it to #gdb on Freenode pointed us to the solution: a parent process had been attached to via “strace -f”. Since only one profiling process can run on any program at one time, the parent process’s “strace”, by following all forks with the “-f”, blocked out our “gdb” from attaching to the child. Simply adding 1 line:

pkill strace
gdb /opt/vendor/redacted/process `pgrep process`

to our debugging solved the mysterious “ptrace: Operation not permitted.” error which was showing me no results in web searches. FYI: this absolutely will block gcore in the same way.

« Previous Page — Next Page »