PCT CA Section K Video
Netbackup 2505 : Semaphores have run out
Issue:
Netbackup database has gone offline. All jobs are reporting a 2505 error. Online community recommends setting semaphores to official recommendations. backupserver already has the recommended settings in place.
Let’s dig into that.
View the current limits: sysctl -a | grep kernel.sem
kernel.sem = 300 307200 32 1024
View the current usage in a very messy manner: `ipcs
Count the currently used semaphores:
ipcs | grep Semaphore\ Arrays -A2000 | grep ^0 | wc -l (yes… there are cleaner ways of doing this- I am not making this pretty though)
This resulted in seeing 1024 currently used (out of 1024)
Stopping the DB resulted in 1015 consumed…
The thought at this point was that either netbackup was not releasing semaphores, or another process was consuming them. I hear they are tasty.
Let’s map semaphores to PIDs- this is a huge hassle to do manually as you have to do a lookup of each semaphore ID to a PID with the -s -i flags (specify semaphore with -s, specify -i for “print details on resource identified by id”)
Can anyone say “FOR LOOP”?
for pid in $( for semid in $( ipcs -s | awk ‘/0x/{ print $2 }’ ) ; do ipcs -s -i $pid | head -9 | tail -n1 | awk ‘{print $5}’ ; done
… this strips the key out of the results of ipcs -s, and then tosses the semaphore ID to ipcs via the specification of -i for a lookup on the specific semaphore. Easy.
We get this fallout: (with the highly sophisticated uniq -c tool…. Sarcasm is enhanced after 6 cups of coffee.)
3 2137
1 32550
5 2137
683 32550
1 36917
3 32550
281 36917
1 16356
6 36917
23 16356
1 6639
3 16356
We see a pattern here: pid 32550 has a bunch open!
So… what is that process?
Given that our max pid is 40960 on backupserver… we could have wrapped around again… (see: `cat /proc/sys/kernel/pid_max`) but if we are to believe what /var/log/messages has to say…
This was ‘nfsidmap[32550]’
The second-most semaphores loving PID was also nfsidmap: nfsidmap[36917]
A no-longer living process never released these. This was likely an issue with nfsidmap
So I rebooted backupserver- satisfied that the mine was still active in our environment… lurking for another day.
Conclusion:
Running backups again…
Only seeing 13 semaphores actively used:
[root@backupserver ~]# ipcs | grep Semaphore\ Arrays -A2000 | grep ^0 | wc -l
13
Story Time: The Server Room is on Fire
The result of a bad draft and a firmware update
2:13am Phone rings with a very concerned person on the other end of the line. “The server room is full of smoke, we think a server is on fire”
In a groggy state, I ask the obvious question: “Is it an emergency? do you want me to come in and check it out?” response: “YES!”
Considering myself to be a daily firefighter- I didn’t think much of the situation, so I drove on the deserted streets to work. Upon quick glance, I did not see any flames from outside the building- which I took to be a good sign.
I get escorted up to the server room by a panicked guard. I ask the reasonable questions of “how is your night going?” to lighten the mood.
I should take a moment to explain… most offices keep their server rooms where the business folks do not have to see or smell the IT folk, but this was a new concept building that believed in highlighting the technology advances in a prominent fashion.
Upon walking into the server room, I notice the smoke monitoring system is indeed alerting on a higher than average smoke concentration. Still no flames visible.
Going off of the basics, I am smelling for a pleasant “magic smoke” essence as I walk on the hot aisle. Much to my dismay, I am unable to locate the scent I was looking for. Instead I am able to get close enough to the air intake to the room to smell it is coming form the roof HVAC units.
A splendid opportunity as I had yet to test my access to the roof. I begin crawling into the access doors of the HVAC units- trying to trace the smell, and eventually follow them to the inlets from the outside.
Looking around- I spot the source: the generators were spewing diesel soot into the air- all part of the regular maintenance.
This morning was a special morning- the wind had been blowing from the West which directed all of the exhaust into the server room. As it turns out- the building management had run a firmware upgrade which reset the generator testing schedule to a time that closely matched 2am.
PCT Video: Washington Sections K and L
Two years late… and two years of hiking combined (due to miserable weather we dropped out and followed up the following year). Enjoy!
PCT Section H-I: Mammoth to Sonora Pass Video
Happy Holidays everyone!
I have finally convinced myself to edit and publish my PCT video from this year (2016).
I hope you enjoy it!
PCT Sections H-I: Mammoth to Sonora Pass
After a bit of a delay, most of which was due to me needing to replace my computer… Here is my summary of CA Sections H and I:
The first destination was a lake called Thousand Island Lake, it was even more beautiful than I had remembered from my last visit here. The vomiting and passing out may have had an impact on my appreciation however. It is still one of the prettier lakes along the PCT so far.
Continuing up the 11,073ft Donohue pass and I came across this guy- frothing at the mouth as he envisioned eating my gear. I may have mentioned this before but I am not a big fan of Marmots. They are large mountain chipmunks that poo everywhere and eat through gear if you leave it unattended.
The route up to the pass was quite full of snow- something that I was just starting to realize would be the theme of this trip. This cup snow can be quite brutal on the ankles and it reflects the sun a little too well for my liking.
Finally at the top, the valley to Tuolumne Meadows can be seen in the distance. I had around 12 miles to do before 5pm so that I could buy some chocolate milk at the store. The route was fairly straightforward until I hit a cliff and had to backtrack a bit and head off to the left of the valley.
This was the end of the snow for the day, but the beginning of wet shoes. The Sierras had a bit of snow get dumped on them the week before I arrived, and then a heatwave went through which made the “seasonal streams” turn into raging rivers. It was as impressive as it was mood killing.
This was the Tuolumne River- something that I REALLY did not want to try to ford. There were bridges built in convenient locations which were appreciated. I was hurting fairly bad at this point due to chafing so I spent a bit of time at the Glen Aulen campground trying to avoid further damage. This is where I donned my swimsuit that I ended up wearing for the rest of the hike. Airflow is a good thing!
I bumped into my first (very) temporary hiking buddy named oldschool. I appreciated the company during the first few larger river crossings. This one was remarkably cold. Come to think of it, most of the river crossings were remarkably cold.
The next Pass was Benson Pass which sits at 10,150ft. The climb up was full of soft snow and the decent was the worst part of this trip. The trail turned into a constant stream which soaked my feet, and I ended up getting lost for an hour.
This is the decent right before the footprints disappeared and the cliffs appeared. Rock climbing is not what I like to do with snow covered ledges, but it seemed a bit safer than following the actual trail. I didn’t take any photos of the bad stuff as I was using my hands to keep myself steady.
After waking up early, I headed out from my camp and found some logs to use for the first couple of stream crossings. I love logs. I also ate some of this onion plant. These are quite spicy and add variation to my diet of poptarts and snickers bars.
The rest of the day consisted of wet, deep, and fast flowing stream crossings and steep snow fields. The trail just sucked at this point. I was surprised that I was still able to average just over 20 miles per day. The Sierras are certainly pretty, but you must pay your dues for the views.
I ended up camping with Little Cricket and Ducky who ended up being great hiking buddies. We hit the 1,000 mile marker at about 10am and took a few photos. It was nice to finally be done with that many miles of California, I still have quite a few more to hike before I finish that state unfortunately. Mosquitoes were bad here.
The rest of this trip was spent at fairly high altitude battling steep snow and a lot of rock fields. I am happy that I had company for this bit as it was frustratingly difficult and exposed. My mood wasn’t the greatest as my tendons in my feet were swollen and I had large blisters.
This trip was a good reminder of how much the trail can suck, and I called it quits as I needed 3 days to heal but didn’t have enough PTO to continue if I took that time off.
My 2014 hiking buddy Mr. President ended up saving the day massively and picked me up, gave me a place to stay, and got me a return flight the next day. I can’t thank him enough and look forward to hiking with him in the (hopefully near) future!
Back on the trail
Two years ago, I dropped off the Pacific crest trail in a mountain town called Mammoth Lakes. I walked back south on the trail after becoming violently ill and took a bus to Reno. I had already decided that I would be dropping out after a couple more hundred miles but this illness cemented my decision.
So now I must finish the sections that I have left- and I am back in mammoth now.
This plane:
This plane:
And these free beers:
Got me here.
As an added bonus I was able to meet up with Big Red whom I met during my 2014 hike. He is a really nice guy who has kept up with the hiking community much better than I have.
Now all I have to do is get supplied and take a bus to the trailhead and hopefully walk a few hundred more miles!
3 Days in Hawaii: The Big Island
I had the opportunity to take a longer weekend and visit Hawaii. This has been a bucket list item and I had a blast!
The hardest part for my planning was deciding what “visiting Hawaii” actually meant to me- and thankfully I had a travel buddy that helped broaden my scope of activities and enjoy the experience quite a bit more!
Overall- the plan was to get the most out of the least amount of time and I feel that we did a great job of hitting everything that we wanted- all while staying withing a decently strict budget. We flew into Kona and rented a car from Enterprise (I highly recommend spending a little more to go with a reputable car rental company). The overall thought process was to move about the island in a clockwise direction.
This resulted in a plan that included:
Friday night
– grab car
– Drive to Lookout – sleep in car
Saturday Morning
– Waipio Valley Hike –early–
Saturday Day
– Farmers Market Honokaa
– Akaka Falls
Saturday evening
– Pāhoa, HI
Saturday night
– Issac Hale Park
– Volcano National Park (night lava viewing)
Sunday Morning
– Thurston Lava Tube
– hike Kīlauea Iki Crater
Sunday Day
– Volcano National Park
Sunday evening
– Jagger museum and Volcano National Park
Sunday night
– Snorkel with Manta Rays
Monday Morning
– Hawaiian Bakery
– Kahaluu Beach Park Snorkeling
Monday Day
– Relaxing
– Hawaiian food
While we missed the snorkeling at Kahaluu, we substituted a snorkeling adventure with Manta Rays! I can’t recommend this enough!
Overall I think that this itinerary fit our plans well and didn’t waste time at resorts or bars and we felt that we had a great vacation in the end.
Here are some more shots from the adventure:
As Promised: long delays between updates.
It may be evident that I have not been updating this blog very much as of late. Don’t worry- I have been having a ton of fun over the past few months, I have just been debating whether or not blogging hurts the natural world.
I have noticed a growing trend of trail trashers- those that are invited into the wilderness by social media to get “the perfect shot” or “tag” the trails with graffiti so they can leave their mark. I am reminded of what I saw in Southern California while hiking down there:
The issue with the perfect-shot people is that photos can end up trashing nature as people leave trails, cut bushes, and straight up modify nature to make it “prettier”. I do not see this as being helpful to the preservation of these resources for future generations. People who tag the trails are just big jerks- I got in an argument with one such person and they felt it was their right to put their marks on signs and trees. That argument ended in them informing me that I need to carry more paint remover as they were more empowered with their efforts. This didn’t make me happy, so I spent some time trying to figure out if my blog actually hurt nature.
I am still debating it- but I have deleted my photography page, and I am now not posting WTA trip reviews unless it has been a substantial time since the last- we don’t need 20 updates on the Rattlesnake trail each day.
So on to lighter news… here are some shots from the past couple months:
As is probably evident- I have been doing a lot of biking, mainly during my on-call rotation as I can explore near work and get some good exercise in (2016 totals so far: 540mi and 23,169 ft gain). My goal of this Winter was to avoid muscle loss due to inactivity- I am planning on going back to the PCT this Summer for a couple weeks and need as much muscle as is possible to be able to hop back into the higher mileage days needed.
Hope all of your worlds are spinning in the correct direction- and remember LNT when you are exploring. Here is a good reminder of what you can do to minimize impact: https://lnt.org/learn/online-awareness-course
Happy Trails, Friends!
