APB for Voice people here. Jump Upgrade to 9.1.2

aaron0011

Doing a jump upgrade from 6.1.4+ or 7.1.5 to 9.1(2). P2V as well (MCS to UCS). Pretty easy procedure.

1. DRS Production Cluster
2. New nodes matching production cluster in VM environment.
- Create VMs with OVA templaces (Change OS to RHEL v3 32-bit in VM settings)
- Install same version as production cluster using bootable ISOs from TAC
Note: NTP and DNS must be working and reachable in lab environment for clean installation of fresh cluster.
3. Restore DRS from step 1 to new VM cluster environment.
4. Reboot cluster: Should have clone of production environment in new VM lab minus licenses at this point.
4. Install Refresh COP file to Pub and Sub nodes.
6. Start Upgrade to 9.1(2) using downloaded ISO from Cisco CCO.
7. Shutdown VM after upgrade to 9 complete via CLI on VM console. Change OS to RHEL v5 32-bit in VM Settings.
8. Boot back up, Apply new v9 licenses
9. DRS v9 Lab cluster
10. Install fresh new 9.1(2) cluster with bootable media. Required step to normalize unaligned partitions from jump upgrade process.
11. DRS from step 9 to new nodes from step 10.

Not bad right? Well I was running into error in database component when upgrading 6.1.5 to 9.1(2) from step 6. Not having a reachable NTP server can cause this but wasn't the issue in my case. Tons of searching on support forums and nothing... well after going though database logs on inactive partition something interesting came up.

file view inactivelog cm/trace/dbl/sdi/installdb_ru.log.err

12:23:09.085 |*ERROR* Error executing "insert into EndUser (allowcticontrolflag,assocpc,building,deletedtimestamp,department,enablemobilevoice,enablemobility,facsimiletelephonenumber,firstname,fkdirectorypluginconfig,fkmatrix_presence,
homephone,mailid,manager,maxdeskpickupwaittime,middlename,mobile,nickname,ocsprimaryuseraddress,pager,passwordreverse,pkid,remotedestinationlimit,site,status,telephonenumber,title,tkuserprofile,uniqueidentifier,userid) values
('T','','',123456,'','F','F','','******','32b66ffb-48d6-c3d5-33c1-1653b28396fb','ad243d17-98b4-4118-8feb-5ff2e1b781ac','XXX XXX-XXXX','*****.Null@*******.com','G***A',10000,'','XXX XXX-XXXX','',
'sip:*****.Null@************.com','','69c4f936f9cdf45f6bbca2570c31215629bb5d6fb97493478b8ff3db6fffbc55','a677593d-d61e-4327-91e1-031bf4bc90ad',4,'',1,'2848','District',1,'f195e958a1652943a1a2c7ab03885743',
'null'): [Informix][Informix ODBC Driver][Informix]Cannot insert a null into column (enduser.lastname).
12:23:16.504 |*ERROR* Error executing "insert into credential"




This particular end user has a last name of NULL, which is a valid SQL command in the Informix database. So when it tried to insert his last name in that field it ran the null command instead and killed the whole database upgrade. Changed his last name in AD to aNULL, synced with CUCM, new DRS and started the process above again and it worked.

If you ever run into problems upgrading CUCM or Unity...make sure you don't have an end user with the name NULL anywhere!!!!!!!!!

bobfromfpl

Wow nuts.. who would've thought. Did you inform Cisco?

aaron0011

Yes, nothing they can do I was told.

shodown

WoW.


You need to get this on a BUG tracker. Did you open a TAC case?

aaron0011

shodown wrote: »

WoW.


You need to get this on a BUG tracker. Did you open a TAC case?

Yes, I did and it was logged. This is the first instance of this happening according to the escalation engineer I spoke with who finally understood what was happening. The first few guys to take the case were a bit clueless. Who knows if it will find it's way into a bug id or not. It's a rare occurrence for sure.

shodown

You can't expect first line support to catch those problems. You prob has to speak to a customer engineer IV to get this solved. Also the higher you go in TAC the less they know about call manager. The first few guys know call manager, then after that you may get a LINUX guy, SQL Guy, and so on.

aaron0011

shodown wrote: »

You can't expect first line support to catch those problems. You prob has to speak to a customer engineer IV to get this solved. Also the higher you go in TAC the less they know about call manager. The first few guys know call manager, then after that you may get a LINUX guy, SQL Guy, and so on.

Agreed but there is a special Drive to 9 TAC designation going on right now. Any case in that queue should have engineers intimate with the upgrade process. I found the errors indicating exact time and where in the process it failed using RTMT instal logs. It took the escalation engineer to use that data to dig up the above from that one specific db log on the inactive partition.

Also an FYI. If a refresh fails starting from 6.1.4+ it will not recover to previous partition. You can't even run a recovery to boot back to previous version. I had to upgrade from 6.1.5 to 7.1.5, then refresh jump upgrade just to have it a fail again. Going from 7.1.5 it actually recovers back to previous version where the logs can actually be accessed. Without an end user with the name NULL going from v6 directly to 9 works fine.

shodown

Yeah these upgrades are a pain in the ASS, but way better that 8.6 upgrade process where we had to have jump servers. I have around 10 of these scheduled until june, no Greenfield's.

aaron0011

shodown wrote: »

Yeah these upgrades are a pain in the ASS, but way better that 8.6 upgrade process where we had to have jump servers. I have around 10 of these scheduled until june, no Greenfield's.

Agree, the jump upgrade is huge progress in going from 6, 7, 8.0 straight to the latest 9.1 build. The fact that you can do it all virtual now is plus. Kudos to Cisco for addressing this.

wintermute000

Without virtual I wouldn't even want to know. Our vendor did 4--> 6 and IIRC they actually sucked all the info out of 4 using a custom tool then repopulated the config into v6 (good thing back then there were no local RGs, E164 routing, SAF, SNR, sip dial plans yada yada).

At least nowadays whatever happens you can always fire up the old VMs again for foolproof rollback.... a lot safer than relying on that backup RAID-1 drive you pulled from the MCS pub chassis.

Thanks for documenting above, good knowledge.

JeanM

Nice catch!

ande0255

Thanks for the post, really good knowledge!