Sometimes it’s nice to know what’s happening under the hood, so let’s talk about how Group Policy is built, by tearing down how to access a particular policy. First, Group Policy is implemented in 2 parts, an LDAP part and a file part, delivered via SMB (CIFS if you’re oldschool) via DFS (Distributed FIle SYstem). Because the DFS part is replicated completely differently than the AD part, there’s a version number for each Group Policy object that’s stored in both places to keep them in sync. Most GPO engines remember the last version they applied by remembering the lowest of the 2 numbers (the LDAP version and the file version in the GPT.INI), if they don’t match.

Let’s talk about the “Default Domain Policy” which everyone will have one of. To find where that policy lives, you have to ask AD. The policy doesn’t actually live in the OU or Domain where it’s linked, so we have to back out the link:
ldap_search_s(ld, "dc=company,dc=com", 2, "(objectClass=organizationalUnit)", gpLink, base, &msg)
We’ll get back something like:

gPLink: [LDAP://CN={6AC1786C-016F-11D2-945F-00C04fB984F9},CN=Policies,CN=System,DC=company,Dc=com;0]

Now, this is a multi-valued array, because multiple GPOs can be linked, in order, to a single OU or Domain or Site. But we only care about this one, so let’s see what’s in it:

ldap_search_s(ld, "CN={6AC1786C-016F-11D2-945F-00C04fB984F9},CN=Policies,CN=System,DC=company,DC=com", 2, "(objectClass=*)", gPCMachineExtensionNames;gPCFileSysPath;displayName;versionNumber, 0, &msg)

That’ll get us the Client Side Extensions (where the work actually happens), and what the file path to the files in the estension are stored, as well as the pretty name of the Group Policy Object:

displayName: Default Domain Policy;
gPCFileSysPath: \\child1.lwtest.corp\sysvol\child1.lwtest.corp\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9};
gPCMachineExtensionNames: [{35378EAC-683F-11D2-A89A-00C04FBBCFA2}{53D6AB1B-2488-11D1-A28C-00C04FB94F17}{53D6AB1D-2488-11D1-A28C-00C04FB94F17}{D02B1F72-3407-48AE-BA88-E8213C6761F1}][{827D319E-6EAC-11D2-A4EA-00C04F79F83A}{803E14A0-B4FB-11D0-A0D0-00A0C90F574B}][{B1BE8D72-6EAC-11D2-A4EA-00C04F79F83A}{53D6AB1B-2488-11D1-A28C-00C04FB94F17}];
versionNumber: 15;

So we have the Default Domain Policy, as desired, but there are a bunch of client side extensions here. It’d be nice to know what they all do generically, without having to inspect each one.
And TechNet delivers on that desire: a list of all Client Side Extensions (in 2010) by GUID for easy reference. Now, I’m writing this, because someone asked where the Password Policy for the domain was stored. Well, that appears to be in: {827D319E-6EAC-11D2-A4EA-00C04F79F83A} Security, which our Default Domain policy applies. So, let’s go find the data!

One of the attributes in the list we last requested was gPCFileSysPath which returned a normal SMB share. If you browse to that share, you’ll see 3 objects:

  • A folder named “MACHINE”
  • A folder named “USER”
  • a file named “GPT.INI”

The GPT.INI will only have 2 lines:

[General]
Version=15

That’s the version number, that you can compare to the “versionNumber” property from the object. If they’re the same, you’re good. If not, your AD isn’t in sync.

In the “MACHINE” Folder are all the Computer Policy settings, and in the “User” folder are all the User Policy settings. Since we were talking about the Password Policy, which is affected on the SAM on the server, it’s a MACHINE setting. If you were to poke through, eventually you’d find this file:

\\domain\sysvol\domain\\MACHINE\Microsoft\Windows NT\SecEdit\GptTmpl.inf

with this data:

[Registry Settings]
MACHINE\System\CurrentControlSet\Services\Netlogon\Parameters\MaximumPasswordAge=4,15

And there’s your password policy, via LDAP and SMB only.

For a bit of additional background, when a computer processes this data, in this order, it will actually only apply CSEs from the gPCMachineExtensionNames that the computer recognizes and has DLLs (or whatever code, if it’s non-Microsoft vendor CSE) that can apply the CSE. This makes it technically safe to put multiple GPOs for multiple Operating sytsems on the same OU structure, knowing that the client computer won’t even bother downloading the files for un-recognized CSEs.

Now, that’s a lot of stuff to type into ldp.exe, how can we make a report on this a bit easier? Well, PowerShell could do it, but one of the products I work on is PowerBroker Open https://github.com/BeyondTrust/pbis-open and https://www.beyondtrust.com/products/powerbroker-identity-services-open/ which includes a CLI for browing ldap called “adtool”. With a bit of bash, we can list out all the group policy objects by name attached to a single OU:

$ cat report-gpos-by-ou.sh
GP=`adtool -a lookup-object --dn "$@" --attr gPLink`;
GPO=`echo $GP | sed -e 's/\[LDAP:\/\///g' -e 's/;[[:digit:]]\]/ /g'`
if [ -n "$GPO" ]; then
echo "";
echo "$OU";
for P in $GPO; do
G=`adtool -a lookup-object --dn "$P" --attr displayName`;
grep -q "$G" /tmp/gpos.txt;
if [ $? -ne 0 ]; then
echo $G >> /tmp/gpos.txt;
fi;
echo "$G";
done;
fi;
$ ./report-gpos-by-ou.sh "OU=Company,DC=domain,DC=com"
OU=Company,DC=domain,DC=com
PBUL Basics
GP-Preferences
$

Our primary lab here at totalnetsolutions.net is a bit space-constrained (RAM constantly 50% overcommited in ESX), so we have it limited to only 2 domains, a parent, and a child. There’s 2 sites, one of them is a “BranchOffice” with only an RODC. This serves us well for the majority of customer cases, especially since the DHCP pool serves the BranchOffice site, which is routed via DSL to the CorpOffice site.

Recently we needed a 3rd domain (2nd child) for testing a particular user/group creation scheme for a customer. Upon completing testing in August, to reduce RAM utilization on the ESX host (and because it makes the AD Schema even more “real-world”, we dcpromo’ed the DC for the 3rd domain, deleting the entire domain. We deleted the objects, and ran the ntdsutil metadata cleanup, like good AD admins. Then we watched the logs fill up with Information events only for a few weeks, and forgot about the episode.

The thing was: the DC in the 3rd domain, because it was short-lived, was left with its DHCP address (it was never a DNS server), so it got added to the BranchOffice site. According to this 2009 Technet AD Troubleshooting Blog entry from Ingolfur Strangeland, RODCs won’t register as the Intersite Topology Generator (ISTG) for the site, but they will perform that role for themselves. (Background on the ISTG.) Because of this, the *new* DC, which was writeable, took over the role of the ISTG for the site, and made sure that all the replication connections went through it, as the only writeable DC in the site.

The problem is, we removed this server. When we removed the writeable DC in the BranchOffice site, there was no other server available to write “I’m now the ISTG” and replicate that setting outbound to the other site(s)… because the only remaining server was an RODC.

We discovered the problem 2 months later (Rob had a baby and wasn’t paying any attention to the lab for a while) when, after creating 1,000,000 new users in the lab, replication was surprisingly slow, but only into the Branch office. So slow that when we joined new computers, they’d boot up with “Service Principal Unknown” errors, and AD users couldn’t log in. In Active Directory Sites and Services we saw that the ISTG Server and Site were both “Invalid”.  This post from 2011 discusses how to move this, but not if it works for RODCs… the good news is that it does:

  1. open the Configuration container, like with adsiedit.msc
  2. Expand CN=Sites, and then the site with the broken ISTG (likely the site with the RODC)
  3. Double-click to open the properties of CN=NTDS Settings
  4. Find the value: “InterSiteTopologyGenerator” and paste in the full DN (from the Configuration Container, not the RootDN) of the RODC
    1. This is the “distinguishedName” value of the CN=<servername>,CN=<sitename>,CN=Sites,CN=Configuration object of the server in the site in question that *should* be the ISTG.
  5. Click “OK” to Save, use ‘repadmin’ or dssite.msc (AD Sites and Services) to force replication and wait 15 minutes (or your own inter-site replication time)

 

Signing RPMs is supposed to be easy:

gpg --list-keys
cat > ~/.rpmmacros < <EOL
%_topdir /home/rob/rpmbuild
%debug_package %{nil}
%_signature gpg
%_gpg_name Rob A <me@totalnetsolutions.net>
%{__gpg} \
gpg --digest-algo=sha1 --batch --no-verbose --no-armor \
--force-v3-sigs --passphrase-fd 3 --no-secmem-warning -u "%{_gpg_name}" \
-sbo %{__signature_filename} %{__plaintext_filename}
EOL
rpm --resign

But there are a lot of caveats:

  1. If you run this on RHEL or CentOS or Scientific Linux 6.x and have an RSA, rather than a DSA, GPG key, any older systems (5.x or 4.x) won’t be able to properly decode the signatures. If you have a DSA key (and only v6+), the --force-v3-sigs is enough.
  2. If you run this on RHEL or CentOS 5.x, you need to remove the “–force-v3-sigs” option from the .rpmmacros line
  3. http://technosorcery.net/blog/2010/10/10/pitfalls-with-rpm-and-gpg/ is a bit wrong – RSA keys don’t always work, DSA do (there’s apparently a RHN KB article about this, if you have support and licenses to read it. I don’t right now).
  4. RHEL / CentOS / Scientific Linux 7 won’t accept (without warning) RPMs signed with weak keys (or weak digests, in some cases, like sha1)

So, based on the systems you’re trying to build your package for, ensure the signing key you’re using and the digest algorithm you’re using, are supported across all the versions you expect to support… or build multiple RPMs, and put them in separate repos for each OS version.

I recently booted up a long-powered-off test system for a customer, and realized it was still CentOS 4.7. up2date gives one of these error messages:


Error: Bad/Outdated Mirrorlist and Baseurl

[Errno -1] Header is not complete.
Trying other mirror.

To fix, first run this to update to a newer version of yum:

for i in \
libxml2-2.6.16-12.6.i386.rpm \
libxml2-python-2.6.16-12.6.i386.rpm \
readline-4.3-13.i386.rpm \
python-2.3.4-14.7.el4.i386.rpm \
python-elementtree-1.2.6-5.el4.centos.i386.rpm \
sqlite-3.3.6-2.i386.rpm \
python-sqlite-1.1.7-1.2.1.i386.rpm \
elfutils-0.97.1-5.i386.rpm \
popt-1.9.1-32_nonptl.i386.rpm \
rpm-libs-4.3.3-32_nonptl.i386.rpm \
rpm-4.3.3-32_nonptl.i386.rpm \
rpm-python-4.3.3-32_nonptl.i386.rpm \
python-urlgrabber-2.9.8-2.noarch.rpm \
yum-metadata-parser-1.0-8.el4.centos.i386.rpm \
yum-2.4.3-4.el4.centos.noarch.rpm
do
rpm -Uvh http://archive.kernel.org/centos/4.9/os/i386/CentOS/RPMS/${i};
done

(note: the order is important for the dependencies, and the trailing “\” prevents bash from spitting out interpretation errors, but makes it readable.)

Then edit /etc/yum.repos.d/CentOS-Base.repo: comment out mirrorlist directives, and in each enabled section add baseurl=http://archive.kernel.org/centos/4.9/os/$basearch, baseurl=http://archive.kernel.org/centos/4.9/updates/$basearch, etc.

Note the change from vault.centos.org to archive.kernel.org/centos – the CentOS Vault is handling RPM headers differently than yum in CentOS 4.9 expects. The Kernel.org archive does not.

Part 2 of the seriesYum Repositories for Corporate Deployments

Part 1 of this series is: Yum Repositories: Background, Terminology and basics.

The most basic setup of a corporate yum repository would be to build a “base” repository for building centralized installs of systems off of the netinstall CD/USB stick provided by your Linux distro. The next basic repository to set up would be an “updates” repository, synced back to the distro, for bandwidth management for scheduled updates. We’ll walk through these 2 setups in order.

Setting up the repository host server

The first thing to determine is *how* you’re going to host your repository.  HTTP is probably the most common, but FTP and NFS repositories can be built as well.  The additional work for an NFS repository is minimal, so we’ll cover that as well as HTTP, using apache 2.2 as the HTTP server.

First, install the NFS and HTTP server components as well as the required reposync utility (this information is exactly the same for an NFS or HTTP apt repository for Ubuntu – a post for another time) and set them to autostart:
sudo /usr/bin/yum install nfs httpd yum-utils
sudo /sbin/chkconfig on http
sudo /sbin/chkconfig on nfs

Next set up the apache server to share the repository. This requires deciding *where* to host the repository, so we’ll use /srv/repo as our base. Create a file /etc/httpd/conf.d/repo.conf with these contents:

Alias /centos /srv/repo

<directory /srv/repo>
Options +Indexes
AllowOverride None
order allow,deny
allow from all
</directory>

And now share out that directory via NFS and restart the servers:

echo '/srv/repo *(ro,async)' >> /etc/exports
service nfs restart
service httpd restart

The last suggestion I’d make is to create a subdirectory for each OS major version and its architectures in the root of your repository heirarchy, which is simple with bash expansion:

mkdir -p /srv/repo/{5,6}

Setting up a “base” repository

This is as simple as copying the contents of the install DVDs or CDs into the repository. We’ll use the CentOS materials for these examples, but Red Hat Enterprise Linux and Scientific Linux work the same.
First, we need to create our repository structure. This walkthrough will only demo RHEL 6 derivatives, but 5.x will work the same. This is the first time to think about which repositories to build. I’ll create 3 total: “base”, “updates” and “internal”. “base” and “updates” will be the distrobution-provided repositories, and “internal” will be for our company’s internal software. The “base” repository actually lives in a folder called “os”, and each repository has subfolders for each architecture that it supports. Therefore, we need to create 3 directories in /srv/repo/6: “os”, “updates”, and “internal”, and each of those gets 2 subdirectories for our supported architectures: “i386” and “x86_64”. (Also simple with bash expansion.)

mkdir -p /srv/repo/6/{os,updates,internal}/{i386,x86_64}

Now copy the files:

sudo mount -t auto -o loop,ro CentOS-6.3-x86_64-bin-DVD1.iso /mnt/cd1/
cp -R /mnt/cd1/* /srv/repo/6/os/x86_64/
sudo umount /mnt/cd1
sudo mount -t auto -o loop,ro CentOS-6.3-x86_64-bin-DVD2.iso /mnt/cd1/
cp -R /mnt/cd1/Packages/* /srv/repo/6/os/x86_64/Packages/

If you have SELinux enabled, and you should, you’ll need to ensure that Apache’s httpd daemon can read these files with the following:

chcon -Rv --type=httpd_sys_content_t /srv/repo

This will set the SELinux context of the directories and all subfolders to “httpd_sys_content_t”, which is the context httpd can read.
At this point, if your server is named “build” you can do an install from the netinstall CD, and point it to “http://build/centos/6/os/$basearch” as the install point, and all packages will be installed from that location. If you do this “base” install with the 6.4 sources, when 6.5 comes out, read the Changing “base” Warning below.
A point of note, for people who come here confused – I haven’t mentioned “reposync” or “createrepo” tools yet – those will be used much later, or not at all. For the “base” repository, all of the repository metadata is included in the DVD copy.

Setting up the “updates” repository

The updates repository needs to be updated constantly, or it becomes meaningless This is the default channel by which the distribution provides security and bug fixes for the software packages they ship. There are 2 means for synchronizing a remote repository: reposync and rsync. Since rsync is more widely used and easier for many people to understand, and because the benefits of reposync aren’t easy for me to find, we’ll use rsync.

The “updates” repository is large – the 64-bit x86_64 “updates” repository in our lab is currently 14GB, and the “os” repository is 5.6. Each additional architecture and version supported will have similar size requirements.

If you’ve already set up the initial structure from the instructions for the “base” repository, skip this paragraph: First, we need to create our repository structure. This walkthrough will only demo RHEL 6 derivatives, but 5.x will work the same. This is the first time to think about which repositories to build. I’ll create 3 total: “base”, “updates” and “internal”. “base” and “updates” will be the distrobution-provided repositories, and “internal” will be for our company’s internal software. The “base” repository actually lives in a folder called “os”, and each repository has subfolders for each architecture that it supports. Therefore, we need to create 3 directories in /srv/repo/6: “os”, “updates”, and “internal”, and each of those gets 2 subdirectories for our supported architectures: “i386” and “x86_64”. (Also simple with bash expansion.)

mkdir -p /srv/repo/6/{os,updates,internal}/{i386,x86_64}

SKIP TO HERE.
Now to pull down the repository updates from a mirror. First choose a mirror from the list for your distribution: CentOS, Scientific Linux, or RHEL (ask your support team).
To make sure your “updates” repository doesn’t go out of sync, schedule a cron job, by creating a file in /etc/cron.daily/ or in /etc/cron.hourly/ (See this warning before syncing hourly), called “reposync” and pasting the rsync commands into it (here’s ours):

#!/bin/sh

mirror="mirror.steadfast.net/centos"
repobase="/repo/yum/"

for vers in 5 6; do

rsync -art rsync://${mirror}/${vers}/os/x86_64 ${repobase}/${vers}/os/
if [ $? -ne 0 ]; then
echo "ERROR getting os files from ${mirror} for ${vers} x64, quitting."
exit 1
fi
rsync -art rsync://${mirror}/${vers}/updates/x86_64 ${repobase}/${vers}/updates/
if [ $? -ne 0 ]; then
echo "ERROR getting updates files from ${mirror} for ${vers} x64, quitting."
exit 1
fi
if [ $vers -eq 5 ]; then
rsync -art rsync://${mirror}/${vers}/os/i386 ${repobase}/${vers}/os/
if [ $? -ne 0 ]; then
echo "ERROR getting os files from ${mirror} for ${vers} i386, quitting."
exit 1
fi
rsync -art rsync://${mirror}/${vers}/updates/i386 ${repobase}/${vers}/updates/
if [ $? -ne 0 ]; then
echo "ERROR getting updates files from ${mirror} for ${vers}, quitting."
exit 1
fi
fi
done

The first rsync took about 2 hours, but subsequent syncs are much faster, since rsync can skip existing files. You’ll notice 2 things: 1) that we have disabled syncing the 32-bit CentOS 6 updates; 2) that we also rsync the “os” repo – for more on this, read the Changing “base” Warning below. Our lab doesn’t have or test 32-bit CentOS 6, and we always test “latest” releases. Also, I set the particular mirror to be a variable, so that it would be easy to change if required.

Setting clients to use your repositories

The easiest way to set a single client to use your newly-setup repo is to simply edit the /etc/yum.repos.d/*-Base.repo file by hand, but that’s slow. I put a fixed repo file in the root of the webserver (in /repo/yum/, named CentOS-Base.repo. In it, I’ve simply commented out the mirror line, and replaced it with a local baseurl:

[base]
name=CentOS-$releasever - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
baseurl=http://build/centos/$releasever/os/$basearch
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-$releasever

#released updates
[updates]
name=CentOS-$releasever - Updates
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
baseurl=http://build/centos/$releasever/updates/$basearch
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-$releasever
enabled=1


This baseurl doesn’t have to be HTTP, since it is a local server and you installed NFS, in that case, use autofs and the file:// URI as:

baseurl=file:///net/build/srv/repo/$releasever/os/$basearch

I haven’t found information that says conclusively if $releasever is properly expanded in that line, but testing shows it works fine in our lab.

Now to test it out on your CentOS 6 client:

wget -O /etc/yum.repos.d/CentOS-Base.repo http://build/CentOS-Base.repo.6
yum clean all
yum upgrade

Now all your yum installs and searches will come from the local repo.

Changing the “base” Source


This is an issue we ran into during testing of this post – you’ll notice all of the rsync scripts pull from “centos/6” not “centos/6.4”, and that we rsync both “updates” and “os” repos nightly.

Which “base” repo you sync against matters. In testing, we initially had the “updates” repo syncing nightly, but not the “os” repo (which was originally staged from a 6.3 DVD. When CentOS 6.4 was released, our “updates” repo shrunk considerably, and we started receiving “Error: Protected multilib versions:” on all updates. This was because “updates” are updates applied against “base”, so they both must be in sync. Therefore, either set up all your systems to sync “6.4” and have no problems, or sync both “base”/”os” and “updates” simultaneously, at the cost of more bandwidth and storage usage.

If you do keep “base” tracking the latest release, your previously-installed systems will continue to upgrade properly – they just will become CentOS 6.4 if they were previously CentOS 6.3. BUT, PXE booting will break. From my colleague:

If you use PXE boot installs, you need to pay attention to when your “base” repo updates so you can copy the appropriate vmlinux and initrd images into the pxe directories.

Staging internally

And a warning about houly syncing.
Most public mirrors will probably blacklist you if you sync hourly. Don’t do that. However, you can set up multiple internal servers, where SERVERA pulls from a public mirror daily, and SERVERB pulls from SERVERA, or even an intermediary. For Example, SERVERA may be the update server your dev lab uses. When patches are verified in dev, you can push updates from SERVERA to a staging location that SERVERB (an SERVERC, etc.) would rsync from, so that production could be “released” to install those patches.

Again: Don’t do hourly syncs to public mirrors.

Series Links

Return to the series header Yum Repositories for Corporate Deployments.

Continue to Part 3 (TBD)

« Previous Page — Next Page »