ITECS Systems Staff Notes

I worked on remedy calls.

I worked on the file server outage on Sunday night.
I helped with the restore from tape on engr00mb on Monday.
I did troubleshooting on the file server to try to determine what went wrong.

I helped with a Realm CRON job that showed that only engr01jab.eos.ncsu.edu
is affected by the missing disk problem in VMware ESX.

I spent time trying to get the O.N. for Solaris patches. I did not have
luck but Richard was able to get them over the phone for me.

I tried to patch Solaris 10 with the newest security patches. But
so far I have had little luck. My security patchlist only install
produces a system that will not boot. I will try downloading more
patches next week to see if the recommended patches fix this no boot
problem. We are still waiting to hear back from OIT if they suggest
any specific patches. It seems we are the only ones on campus using
Solaris 10 for file servers. Evidently OIT is still using Solaris 8
for all AFS servers. So it may take a bit for them to figure out what
we may need to add. But for now I am using sunsolve to try to get the
machines pactched up to the current release of Solaris in case that helps.
(It also will make the machines more secure)

I upgraded kernels on all Linux lab / vcl / desktop type systems.

I upgraded kernels on all Linux ESX boxes I am responsible for. I also
removed the clock=pmtmr argument from the grub.conf file on all ESX
servers I upgraded.

I went to a CLS services meeting. I am supposed to work with Jack Neely
to make some VMware features we use be a part of Realm Linux globally in
the near future.

I created a "findutils" rpm for Solaris 10. This was to add the "locate"
command on Solaris 10.

I finished all the new proe packages for Solaris 10.

I tried to contact Steven Stewart in OIT about them taking over our
backups. He still hasn't replied to either of the emails I sent him.

I nagged OIT about the printing issues again this week and they FINALLY
pushed out the update...

Mediasite
-worked on SAN backups w/ Steven
-worked on ftp setup on new EX server
-investigate SQL hotfix upgrade errors.
-worked on AD LDS backups

Other
-Mediasite Planning Meeting
-Discussed license log moving w/ Michael
-Email discussion w/ Michael

• mon/tues
  • worked to restore mail server
• wed
  • cls meeting and fixing boyette's research lockers
• thurs
  • 2 educause sessions
  • provisioned a new vhost
  • worked more on boyette's lockers
  • discovered jabber server drives were ro and worked on a script to search out other machines with the same problem
• fri
  • fixed a problem with the vhost i created
  • turns out that script won't work, cron can't create a temp file to send mail if it finds a ro partition
  • rebooted the jabber server to clear error conditions and applied the new cert

Activities:

• Completed Engineering vCenter maintenance and documentation
• Worked with Daniel on mail server log issue and resolution
• Worked with Gary on tape rotation issue that I was responsible for
• Worked out some logistics in 110 Poe with Design

Meetings:

• VMware team meeting
  • Discussed vCenter backend DB
• Met with IES about VMware migration roadmap
  • Should be getting us the first two servers the week of Nov 9th
  • Planning on being off all old servers by Christmas
  • Possibly another 2950 available then
• Webinar on Groundwork Monitor 6
• Webinar on VMware Orchistrator
• View naming standards with the Web Team
  • Damian should be writing it up in the helpdesk wiki
• webhosting pilot meeting

Started looking at reworking the domain level baseline policies. To this end, i searched for a (preferably free) tool that would diff two group policies, as this would make things a bit easier. Tried one, but gave errors when used. Also came across "Microsoft Advanced Group Policy Management (AGPM) that helps you better manage Group Policy objects (GPOs) in your environment by providing change control, offline editing, and role-based delegation." Downloaded and read up on this a bit, perhaps for a future discussion.

Page 5 HP4100 printer seems to be failing. Worked on diagnosing the error on the LED display, and recearch indicates it might be a bad ROM SIMM. Tried to figure out if there was a way to flash the firmware; no luck. Looked at current HP offerings for a replacement printer.

Worked on DC full backup scripts that backup to a network location. Still not working right.

Worked with Billy to get the Nagios monitoring client installed on hte WolfTech domain controllers, and configured the OIT Nagios server to monitor them. Took a look at the 1.4 version of the "check_ad.exe" component of the Nagios client package, and asked mpunderw to build me another msi with the updated client in it. Want to test this in the WolfTest domain before trying in WolfTech.

Patched print servers.

- checked on Solidworks and Windows 7 compatibility
- tested all lab software against Windows 7
- worked on trying to get a new Windows 7 compatible AFS client working
- admindesktop migration
- help Sherry in BAE track down a software deployment issue

• Fixed the log problem on engr00mb. Turns out postmaster was over quota and it was bouncing stuff like crazy. Created a quota monitor cron job on the local box to check for any over 95% and email me, Richard, and sysrootmail with a warning.
• Elockers and server write stuff as usual.
• Watched an interesting Groundwork Monitor webcast.
• Talked to Darren from Comtech about logging/splunk and the presentation in RTP in a few weeks. Going to gather some data from him about our daily log size and see if we want to join in on a splunk license.

I changed the backup tapes.
I worked on Remedy calls.

I did a lot more testing with the VMware VMXNET 2 driver disks. All ISOs
on the ESX and ESXi servers have been replaced with my custom ISOs with the
proper drivers on them. Everything is ready to create new VMs with VMXNET 2
on Realm Linux.

I changed the "batch-ping" command so that it now pings a group of
machines twice. The first sweep is to "wake up" all the PCs in the list
and the second is the actual ping used in emails / output. This change
was requested by Justin Lancaster and is now live.

I did some research on power saving on Linux desktop systems and I also
consulted with Jack Neely on the problem as it was presented to me. He
is going to give this problem more thought for RHEL 6.

I created the openoffice-nautilus-integration rpm for Solaris 10 to
fix a bug with nautilus not knowing how to start a openoffice icon on
the desktop.

I created the openpkg-perlfix rpm to fix a flaw with the openpkg system
not being able to package perl scripts. This fix is required to
re-package the proe wildfire rpms.

I upgraded the license managers data file on engr14lic and engr15lic at the
request of Robbie Little to add a new feature they need.

I started working on a new set of packages for proe wildfire. I am trying
to get openpkg to work with "subpackages" which would be a more proper way
to package this massively huge app.

I tried to get OIT to push out a fix for the Linux printing problems. It
looks like OIT still has not pushed that rpm out to the labs. I did
determine that the new version of cups2lprng fixes the problems.

Mediasite
-Updated Firewall for Biltmore live presentation next week
-Discussed recorder for History department w/ DELTA
-Troubleshooting History recorder connection to instance
-Reinstalled Media Server Control Service on engr99wms
-Set up IES for Greensboro publishing test.
-fixed authentication timeout in management portal (now 120 minutes)
-Got to the bottom of particularly annoying problem: Mediasite not recognizing Wolftech group membership of a newly created role.
-ran storage space statistics
-worked on Veritas NetBackup w/ Steven for clustered files

Other
-ITECS Managers Meeting
-COE Computer Committee
-Wolfwise Town Hall

• Reinstalled laptop with Win7 Pro, had Billy join it to the domain
• Worked on futureshock/rhel5 web kit
• Processed lockers and web write requests
• Engr-sysadm meeting
• Assisted Pete with some log server stuff
• Played around with OIT's splunk setup

- tried to get groupwise installed on Windows 7
- tested whether or not lab programs would run in Windows 7
- worked on Admindesktop migration
- looked at default domain policy for Windows 7

Looked at some suspect GPOs noted by the output of gpotool jaklein ran.

Wrote a batch file that uses several of the GPMC scripts to gather various information about, and then backup the WolfTech domain group policies. Also worked on a script that performs a full system backup of a domain controller to a network share. Still need to finalize the expiring of system backup jobs, though.

Discussed DC backups with Billy; sent out some notes to the WolfTech domadmin list.

Worked with Billy to install and configure the Nagios client (packaged by jaklein) onto the WolfTech domain controllers. Went thru how to setup client on the OIT server monitoring webpages.

Worked on a LogParser script that will record reboots and shutdowns of key WolfTech central servers.

Worked on a LogParser script to gather the names of those computers "in" the domain with broken trusts.

Tweeked the WolfTech DC Health Report script.

I'm gone for the next 3 weeks in a foreign land full of wonderment and shenanigans. Sucks to be everyone else.

• TMOS 10 - Got a gpo setup, it will update v8, but its talking to the wrong parent server.
• ENGR Cleanup - Deleted krb_disable user accounts and removed a couple more GPO/OU/groups.
• Project Planning - Went around to everyone in Systems and a couple people on the second floor to discuss what would be going on for the next couple weeks to help them prioritize and to give feedback on anything that would normally require my assistance/input/etc.
• Networking - Troubleshooting Celerra access issues, going through QIP DB dump to try and distinguish VLANs from subnets, monitoring the DC's via OIT's nagios on Sysnews.
• Podcasts - Got some MS, FLOSS, VMware, Security podcasts to fill up the 60+ hours of travel time I'll have to/from New Zealand.

Meetings:

• IAM Roadmap
• Solar Center -> Wolftech
• Engr-Sysadmin
• Domain Controller Monitoring
• AD Policy Committee
• Project Planning x 6
• Network Architect Interviews x 3

I worked on remedy calls.
I changed the backup tapes.
I went to the Engineering sysadmin meeting.

I pushed out a new openoffice.org to the labs. The default version users
see is now dependent on which openoffice-local rpm is installed rather
than which version in AFS is the default. The app does better apploging on
Linux now. Most of the time it will show oocalc or oowriter instead of
the generic ooffice as the app that was launched by the user.

I created a new rpm called openoffice-local-bin which does a lot of the
work. Both rpm packages will work on a stock RHEL 5 or CentOS 5 box. Thus,
I have made the repository world readable. I also created documentation
on this on the techies wiki here.

The packages I built for openoffice.org 3.1.1 are much better than earlier
versions of the rpms I had created. They do all the work in the %build
section rather than the %post section.

The better openoffice-local package now will install on Solaris 10
at install time rather then needing to be added post install. I'm pretty
sure this is because the packages are more properly built now.

I started working on better versions of the rpms for proe wildfire so
that they will also be able to install at install time on Solaris 10.
The rpms are working but they have exposed some other bugs in Solaris 10.
So this isn't ready to share with the Solaris 10 list yet. I will need
to spend more time on this problem next week.

I finished all the ESX VMXNET 2 RHEL 5 Realm Linux boot ISOs and I moved
them onto each ESX box. I created a method to build the VMware tools
modules post install since we don't have AFS on first boot.

I still need to do a bit more testing but this is basically finished now.

I updated this article which describes different issues that can come up
running Realm Linux within VMware. The article had become out of date.

I modified the include "ESX-vmconfig" within RHEL 5 so that it no longer
adds "clock=pmtmr" to grub.conf since its no longer required. I also
changed this file for RHEL 4 to add the "divider=10" parameter in
case we ever do any more RHEL 4 installs.

I created a spreadsheet with everything I think OIT needs to begin taking
over our backups. I am waiting to hear back from Steven Stuart in OIT to
see what the next steps we need to take are.

Mediasite
-Troubleshooting login issue for Sharon B of DELTA (extensively)
-Setup FTP, EX Server, IIS, and NetBackup on engr97ex
-Windows Patches Applied
-Research on SQL Failover Cluster hotfix
-More work with AD LDS than I care to think about. (fixing replication, failover)
-crosstraining Michael

Other
-Engr Sysadmin Meeting
-TracePro & License Server Space w/ Michael

Installed OIT based certs on the Wolftest domain controllers. Q291010

Researched a little on the "Windows NT compatibility mode" secure channel mechanism in regards to Macintosh connectivity to SMB shares. Macs apparently still use antiquated cryptography algorithms to establish secure channels with AD resources, and Apple's solution is to downgrade AD's supported cryptography algorithms.
http://support.apple.com/kb/TS2967
http://support.microsoft.com/kb/942564

Created service account for ADToolKit to unlock WolfTech accounts.

Worked on getting Terminal Server Licensing established for engr99ts with kmswann2. Was not successful. Rebuilt engr99ts and reinstalled apps, giving us another 120 days to figure it out.

Installed the Windows 7 Enterprise Client beta Solution Accelerator GPO templates; first in WolfTest, then in WolfTech.

Updated permissions on the ENGR print server to only allow ENGR domain users to print to it, thus blocking inappropriate access of WolfTech accounts due to the recent creation of an AD trust created with it for migration purposes to WolfTech. Also updated the "COEDEAN-COEDEAN-Staff-Printing" domain group policy setting for the WolfTech COEDEAN staff print server OU, ensuring that only "COEDEAN-Staff" can print to the WolfTech COEDEAN print server.

Installed Win2008/R2 on machine "engr98pt" to support 64 bit COEDEAN Windows clients.

Tweaking of the AD_Objects log parser script.

Futzed with M$ SQL Server to see if i could figure out how to get it to email any SQL Backup error messages. Figured out the SMTP mechanism, but not how to trigger email notifications on failures.

- Finished lab image v3.1
- began deploying new lab image.

• Networking - Troubleshooting moving VIPs on vlan 30 vs. other VLANs and how that interacts with VMware internal networking. Getting changes made to the Research IV firewall for ITRE. Diagnosing communication issues with VLAN 25->Celerra.
• ENGR - LOTS of cleanup, including replacing 15 WSUS policies with 1 using GPP targetting, removing a bunch of unused/redundant GPOs, unused groups/users/computers. Setting up some application upgrades.
• Wolftech - Discussion of application upgrade options, authentication issues with samba+Win7, and encryption for mac shares. Working on requirements for the ad object auditing report. Creating trusts for engr/itre. Diagnosing issues with the ou creation script.
• Mediasite - Helping with some troubleshooting and cert procurement.
• Logs - Trying out OIT's splunk setup and asking Pete lots of questions about his work.
• IES - Fixing DNS entries and diagnosing AD replication issues in the IES AD domain.

Meetings:

• ITECS Managers
• CSC + VMware
• AITD
• OIT Clustering
• Security Subcommittee
• ISE Consulting
• WolfWise Customer Advisory Team

Activities:

• Troubleshooting the Career Fair website and why it caused issues for the webserver
• Started investigating Storage vMotion tools
• Worked with Daniel on futureshock config
• Research on vCenter maintenance tasks
• Working with VMware on some permissions model issues
• Network troubleshooting in vlan 30. Seems like moving vIPs around doesn't update ARP tables appropriately

Meetings:

• Met with IES about server migration plans
• Met with IES to go over VM provisioning
• VMware webcasts
• Met with Patrick to discuss current VMware cluster needs, cluster merging, and partnership with OIT
• Campus Web Developers meeting
• Webhosting Pilot Q&A session
• Webhosting Pilot project team meeting

• 64 bit webserver kit is functioning, working on tweaking
• shibboleth meeting
• security subcommittee meeting
• working with pete on log server, trying to get rsyslog working right
• elockers and write requests

- lab image
- deployed firefox and quicktime in engr

Mediasite
-Discuss Upgrade Plans w/ Sonic Foundry
-Deb Manzo email question
-Load Balancer / DNS question
-130 Park Shops live publishing point
-3170 Broughton Troubleshooting / Nuttiness
-Prep for 5.1 Upgrade
-engr97ex setup work
-Perform 5.1 upgrade

I was able to finally setup the LogParser scripts to run periodically, dumping the data into the My SQL database. Ran into problems with the commandline utility "CSV2MySQL.exe", as it would generate errors when run on the cron server, and there is no documentation on it. Figured out two piees of middleware that needed to be installed, and it began working. Got some email reports. Setup some scripts to help me monitor the log gathering process. Also discussed an "account lock/unlock" piece that i will implement. Stein's created the table, and just have to start dumping the data into it.

Checked on the domain export/import results. There were some errors, but i believe it was generally successful. Some of the errors were expected, but some, i believe, were due to commas, apostrophes, and slashes in object names, which interferes with the script's syntax/delimiter notions.

Went to Mac/AD permission meeting.

Went to Shibboleth & Federated Identity Management lunch and learn.

• ENGR - Firefox upgrade, Quicktime upgrade, and deleted almost 200 objects. More cleanup needed.
• Mediasite - Did some diagnosis on the issues with one of the EX nodes and then got comtech to pull it from the load balancer.
• Architect - More reviewing resumes.
• FF+GW - Diagnosing the Firefox upgrade interaction with Groupwise. Turns out, if you don't have a default web browser set, the Groupwise spell checker crashes GW. Awesome.
• ITRE Firewall - Working with Comtech to get it setup correctly so that we can setup a one-way AD trust.

Meetings:

• IAM Conf Call
• IES VMware 1
• Shibboleth
• NDSTech
• Mac AD Permissions
• IAM Service Team
• IES VMware 2
• Download.eos