Puppet, FreeBSD and custom $PACKAGESITE

Joar Jegleim Leave a comment

I recently noticed a puppet managed server installed a package that wasn’t supposed to be available …
The thing is, we’re running Tinderbox to build our own packages from ports, it’s kind of normal when using FreeBSD and it’s ‘rolling release, source based software package system (ports)’ .
Anyway, we’ve setup all servers with a global PACKAGESITE variable pointing to our local repo, so that pkg_add and portmaster will pull packages from there.
We distribute config files and so on via puppet, and what happened was that when running puppet initially it will push out the global /etc/csh.cshrc config which contains the magic PACKAGESITE setup, but at the same time it will start installing a bunch of packages without actually source’ing the /etc/csh.cshrc … So actually PACKAGESITE is empty the first time puppet runs, and that means pkg_add will default to the official FreeBSD packagesite …
This is not how I want it, because for instance a FreeBSD 8.3-RELEASE with an empty PACKAGESITE will pull packages from a spesific ports ‘freeze’ around the 8.2-RELEASE, and that’s a long time ago today … Next time I install any extra packages on that system, it’ll use our internal PACKAGESITE, and now our system has a mix of our lates ports and the original ports from RELEASE, and stuff start to get complicated ….

Ruby, bundle install –without development test might be troublesome in FreeBSD

Joar Jegleim Leave a comment

I’m setting up a new Redmine installation on a FreeBSD server. I’m having trouble with:

# bundle install --without development test
Installing sqlite3 (1.3.7) with native extensions 
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.
...

Turns out I have to install the sqlite3 gem by giving bundle a hint on where to find the sqlite3.h file .

# gem install sqlite3 -- --with-sqlite3-include=/usr/local/include
Building native extensions.  This could take a while...
Successfully installed sqlite3-1.3.7
1 gem installed
Installing ri documentation for sqlite3-1.3.7...
Installing RDoc documentation for sqlite3-1.3.7...

FreeBSD upgrading packages with portmaster

Joar Jegleim Leave a comment

So we’re running our local package repository for our FreeBSD farm. When using FreeBSD, can choose to stick to binary packages for your current RELEASE (which never get patched for security holes), or you could use binary packages for the stable branch of your RELEASE (they’ll be patch’ed for security holes, but they’ll also keep rolling into new versions, e.g. if you got a 100 or so servers you kinda fucked if you want to have consistency with versions of packages on your servers), or you could, like us (and many other FreeBSD admins) run your local repo.
With a local repo you get control on versions of packages, at least, as a bonus you’ll also get all kinds of freakin’ dependency problems, tinderbox doesn’t always cooperate, and so on.
So we configure portmaster on each server to use binary packages only, and point to our package repo server . Then you’ll ‘only’ have to issue portmaster -a to upgrade your packages on a server… I’d say you got about 30% chance of that working out for you on your first run .
Here’s what I usually have to do (kinda pseudo’ish cli):

portmaster -a
(portmaster brakes on package a ... so manually remove that package so that portmaster -a could finish, then reinstall the troublesome package)
pkg_info | grep package a
pkg_delete package a
ERROR: package b depends on package a
pkg_delete package b
pkg_delete package a
portmaster -a
(portmaster brakes on package c ... so manually remove that package so that portmaster -a could finish, then reinstall the troublesome package)
pkg_info | grep package c
pkg_delete package c
ERROR: package d depends on package c
pkg_delete package d
ERROR: package e depends on package d
pkg_delete package e
pkg_delete package d
pkg_delete package c
portmaster -a
(let's say this time portmaster managed to complete)
pkg_add -r [all the packages you manually removed]
(now I suddenly got a)
pkg_version: corrupted record (pkgdep line without argument), ignoring
(google it, mabye you get <a href=http://www.cyberciti.biz/faq/pkg_version-corrupted-record-pkgdep-line-without-argument-ignoring/>here</a> so you try)
portmaster --check-depends
you'll mabye reinstall any troublesome packages, and if you're lucky you're done

It’s probably because I got more than 10 years of experience in using Debian, and only about 6 months experience in using FreeBSD that I keep getting myself into these kind of problems, but I’m still getting grumpy about it !

semget errors in FreeBSD Jail’s

Joar Jegleim 2 Comments

We got some semget errors when trying to start uwsgi inside a FreeBSD jail . The solution is to set some /boot/loader.conf variables

kern.ipc.shmmni="512"
kern.ipc.semmni="512"
kern.ipc.semmnu="512"
kern.ipc.semmns="1024"
kern.ipc.semume="512"

But we still had some problems in the jail after reboot . It turns out FreeBSD sysctl variable security.jail.sysvipc_allowed defaults to 0, that is ipc by default is disabled for jails. Note that issuing:

# sysctl security.jail.sysvipc_allowed=1

and restarting the jail isn’t enough, ’cause /etc/rc.d/jail will reset that variable upon jail restart, you’d also have to have this in /etc/Rc.conf

jail_sysvipc_allow="YES"

and you’re able to start jails with ipc support .

This is explained in more detail at www.freebsddiary.org
Wonder what all these cryptic sysctl variable names mean ? try:

# sysctl -d kern.ipc.shmmni
kern.ipc.shmmni: Number of shared memory identifiers
#

Also note that: I ran into an other problem where a lot of ipc semaphores where taken but not released, I wrote a small script to fix that:

#!/bin/sh

ipcs | grep [username] | awk '{print $2}' > sem.txt
for i in `cat sem.txt`; do 
 ipcrm -s $i; 
done;

substitute [username] with the username of the users semaphores you’d want to delete .

bind9 using forwarders

Joar Jegleim Leave a comment

I installed bind9 at localhost so that I could setup forwarders for our LAN dns servers + for my Unix dns servers in DMZ ( I don’t have access to configure the LAN dns server, which would be a better option) . I setup something like this in /etc/bind/named.conf.local (Debian type of distro):

zone "windows.domain.servers." IN {
   type forward;
   forward only;
   forwarders { 10.0.0.1; 10.0.0.2; };
};

zone "unix.dmz.servers." IN {
   type forward;
   forward only;
   forwarders { 192.168.0.1; };
};

domain names and forward ip’s has been changed .
With this setup, I could configure my workstation to use localhost for dns queries, and tell /etc/resolv.conf to search for both the domains I want regular access to without typing the fqdn .
But there was a problem getting answers from my unix dns server in DMZ, /var/log/syslog told me something like:

Nov 15 09:27:07 uranus named[8184]:   validating @0x7f94784dgd70: unix.dmz SOA: got insecure response; parent indicates it should be secure
Nov 15 09:27:07 uranus named[8184]: error (no valid RRSIG) resolving 'lb01.unix.dmz.servers/DS/IN': 192.168.0.1#53
Nov 15 09:27:07 uranus named[8184]: error (no valid DS) resolving 'lb01.unix.dmz.servers/A/IN': 192.168.0.1#53

Since I haven’t got time to debug this stuff in detail, I simply edited /etc/bind/named.conf.options and setup

dnssec-validation no;

And my localhost bind9 could resolv stuff from our DMZ
Thnx isc.org and
groups.google.com

rsync and variables

Joar Jegleim Leave a comment

I ran into this annoying thing about rsync when I was writing this new backup script in BASH . I wanted support for excluding directories in config files for clients being backed up, since rsync support –exclude=’some/path’ I could have config files with stuff like this:

EXCLUDE="--exclude='proc/*' --exclude='dev/*'"

But while testing this I noticed rsync simply ignored the EXCLUDE variable, so I created this little test where I put in some arbitrary proc and dev directories with some arbitrary sub-directories, check out this example:

joar@uranus:~/tmp$ rsync -va --delete --exclude "dev" --exclude "proc/*" rsynctest/ rsynctest2/ | grep proc
jdk1.7.0_07/jre/lib/amd64/libsaproc.so
jdk1.7.0_07/proc/
proc/
joar@uranus:~/tmp$ EXCLUDE="--exclude 'dev' --exclude 'proc/*'"
joar@uranus:~/tmp$ rsync -va --delete $EXCLUDE rsynctest/ rsynctest2/ | grep proc
jdk1.7.0_07/jre/lib/amd64/libsaproc.so
jdk1.7.0_07/proc/
jdk1.7.0_07/proc/1
jdk1.7.0_07/proc/2
jdk1.7.0_07/proc/3
jdk1.7.0_07/proc/4
jdk1.7.0_07/proc/5
jdk1.7.0_07/proc/6
proc/
proc/1/
proc/2/
proc/3/
proc/4/
proc/5/
proc/6/
proc/7/
proc/8/
joar@uranus:~/tmp$

Notice when I specify –exclude= in the first rsync, the proc dir is rsync’ed without sub-dirs, but when i put that stuff into the EXCLUDE variable the subdirs of proc is being rsynced (!) . I poked around this for a while trying double quotes VS. single quotes, even tried expansion stuff like $(echo $EXCLUDE) .
After a good night sleep I came up with this workaround:

joar@uranus:~/tmp$ EXCLUDE='proc/* dev/*'
joar@uranus:~/tmp$ for f in $EXCLUDE; do echo $f >> tmpfile; done
joar@uranus:~/tmp$ cat tmpfile 
proc/*
dev/*
joar@uranus:~/tmp$ rsync -va --delete --exclude-from=./tmpfile rsynctest/ rsynctest2/ | grep proc
jdk1.7.0_07/jre/lib/amd64/libsaproc.so
jdk1.7.0_07/proc/
proc/
joar@uranus:~/tmp$

So I got what I want πŸ™‚
The reason I want to cat stuff from the EXCLUDE variable into a tmpfile is because I want 1 config file pr. backup client. Those who setup new backups shouldn’t have to remember to setup a 2nd config file with directory exceptions.