Netbackup 2505 : Semaphores have run out

Issue:

Netbackup database has gone offline. All jobs are reporting a 2505 error. Online community recommends setting semaphores to official recommendations. backupserver already has the recommended settings in place.

Let’s dig into that.

View the current limits: sysctl -a | grep kernel.sem
kernel.sem = 300 307200 32 1024
View the current usage in a very messy manner: `ipcs
Count the currently used semaphores:
ipcs | grep Semaphore\ Arrays -A2000 | grep ^0 | wc -l (yes… there are cleaner ways of doing this- I am not making this pretty though)

This resulted in seeing 1024 currently used (out of 1024)

Stopping the DB resulted in 1015 consumed…

The thought at this point was that either netbackup was not releasing semaphores, or another process was consuming them. I hear they are tasty.

Let’s map semaphores to PIDs- this is a huge hassle to do manually as you have to do a lookup of each semaphore ID to a PID with the -s -i flags (specify semaphore with -s, specify -i for “print details on resource identified by id”)

Can anyone say “FOR LOOP”?

for pid in $( for semid in $( ipcs -s | awk ‘/0x/{ print $2 }’ ) ; do ipcs -s -i $pid | head -9 | tail -n1 | awk ‘{print $5}’ ; done

… this strips the key out of the results of ipcs -s, and then tosses the semaphore ID to ipcs via the specification of -i for a lookup on the specific semaphore. Easy.

We get this fallout: (with the highly sophisticated uniq -c tool…. Sarcasm is enhanced after 6 cups of coffee.)

3 2137
1 32550
5 2137
683 32550
1 36917
3 32550
281 36917
1 16356
6 36917
23 16356
1 6639
3 16356

We see a pattern here: pid 32550 has a bunch open!

So… what is that process?

Given that our max pid is 40960 on backupserver… we could have wrapped around again… (see: `cat /proc/sys/kernel/pid_max`) but if we are to believe what /var/log/messages has to say…

This was ‘nfsidmap[32550]’

The second-most semaphores loving PID was also nfsidmap: nfsidmap[36917]

A no-longer living process never released these. This was likely an issue with nfsidmap

So I rebooted backupserver- satisfied that the mine was still active in our environment… lurking for another day.

Conclusion:

Running backups again…

Only seeing 13 semaphores actively used:

[root@backupserver ~]# ipcs | grep Semaphore\ Arrays -A2000 | grep ^0 | wc -l

13

Advertisements

Story Time: The Server Room is on Fire

The result of a bad draft and a firmware update
2:13am Phone rings with a very concerned person on the other end of the line. “The server room is full of smoke, we think a server is on fire”

In a groggy state, I ask the obvious question: “Is it an emergency? do you want me to come in and check it out?” response: “YES!”

Considering myself to be a daily firefighter- I didn’t think much of the situation, so I drove on the deserted streets to work. Upon quick glance, I did not see any flames from outside the building- which I took to be a good sign.

I get escorted up to the server room by a panicked guard. I ask the reasonable questions of “how is your night going?” to lighten the mood.

I should take a moment to explain… most offices keep their server rooms where the business folks do not have to see or smell the IT folk, but this was a new concept building that believed in highlighting the technology advances in a prominent fashion.

Upon walking into the server room, I notice the smoke monitoring system is indeed alerting on a higher than average smoke concentration. Still no flames visible.

Going off of the basics, I am smelling for a pleasant “magic smoke” essence as I walk on the hot aisle. Much to my dismay, I am unable to locate the scent I was looking for. Instead I am able to get close enough to the air intake to the room to smell it is coming form the roof HVAC units.

A splendid opportunity as I had yet to test my access to the roof. I begin crawling into the access doors of the HVAC units- trying to trace the smell, and eventually follow them to the inlets from the outside.

Looking around- I spot the source: the generators were spewing diesel soot into the air- all part of the regular maintenance.

This morning was a special morning- the wind had been blowing from the West which directed all of the exhaust into the server room. As it turns out- the building management had run a firmware upgrade which reset the generator testing schedule to a time that closely matched 2am.

PCT Sections H-I: Mammoth to Sonora Pass

After a bit of a delay, most of which was due to me needing to replace my computer… Here is my summary of CA Sections H and I:

Thousand Island Lake Pano sml

The first destination was a lake called Thousand Island Lake, it was even more beautiful than I had remembered from my last visit here. The vomiting and passing out may have had an impact on my appreciation however. It is still one of the prettier lakes along the PCT so far.

DSC08618edsml

Continuing up the 11,073ft Donohue pass and I came across this guy- frothing at the mouth as he envisioned eating my gear. I may have mentioned this before but I am not a big fan of Marmots. They are large mountain chipmunks that poo everywhere and eat through gear if you leave it unattended.

DSC08602edsml

The route up to the pass was quite full of snow- something that I was just starting to realize would be the theme of this trip. This cup snow can be quite brutal on the ankles and it reflects the sun a little too well for my liking.

DSC08646ed2sml

Finally at the top, the valley to Tuolumne Meadows can be seen in the distance. I had around 12 miles to do before 5pm so that I could buy some chocolate milk at the store. The route was fairly straightforward until I hit a cliff and had to backtrack a bit and head off to the left of the valley.

Untitled_Panorama2edsml

This was the end of the snow for the day, but the beginning of wet shoes. The Sierras had a bit of snow get dumped on them the week before I arrived, and then a heatwave went through which made the “seasonal streams” turn into raging rivers. It was as impressive as it was mood killing.

DSC08724edsml

This was the Tuolumne River- something that I REALLY did not want to try to ford. There were bridges built in convenient locations which were appreciated. I was hurting fairly bad at this point due to chafing so I spent a bit of time at the Glen Aulen campground trying to avoid further damage. This is where I donned my swimsuit that I ended up wearing for the rest of the hike. Airflow is a good thing!

DSC08732edsml

I bumped into my first (very) temporary hiking buddy named oldschool. I appreciated the company during the first few larger river crossings. This one was remarkably cold. Come to think of it, most of the river crossings were remarkably cold.

DSC08760edsml

The next Pass was Benson Pass which sits at 10,150ft. The climb up was full of soft snow and the decent was the worst part of this trip. The trail turned into a constant stream which soaked my feet, and I ended up getting lost for an hour.

DSC08791edsml

This is the decent right before the footprints disappeared and the cliffs appeared. Rock climbing is not what I like to do with snow covered ledges, but it seemed a bit safer than following the actual trail. I didn’t take any photos of the bad stuff as I was using my hands to keep myself steady.

After waking up early, I headed out from my camp and found some logs to use for the first couple of stream crossings. I love logs. I also ate some of this onion plant. These are quite spicy and add variation to my diet of poptarts and snickers bars.

This slideshow requires JavaScript.

The rest of the day consisted of wet, deep, and fast flowing stream crossings and steep snow fields. The trail just sucked at this point. I was surprised that I was still able to average just over 20 miles per day. The Sierras are certainly pretty, but you must pay your dues for the views.

DSC08865edsml

I ended up camping with Little Cricket and Ducky who ended up being great hiking buddies. We hit the 1,000 mile marker at about 10am and took a few photos. It was nice to finally be done with that many miles of California, I still have quite a few more to hike before I finish that state unfortunately. Mosquitoes were bad here.

This slideshow requires JavaScript.

The rest of this trip was spent at fairly high altitude battling steep snow and a lot of rock fields. I am happy that I had company for this bit as it was frustratingly difficult and exposed. My mood wasn’t the greatest as my tendons in my feet were swollen and I had large blisters.

This trip was a good reminder of how much the trail can suck, and I called it quits as I needed 3 days to heal but didn’t have enough PTO to continue if I took that time off.

My 2014 hiking buddy Mr. President ended up saving the day massively and picked me up, gave me a place to stay, and got me a return flight the next day. I can’t thank him enough and look forward to hiking with him in the (hopefully near) future!

Back on the trail

Two years ago, I dropped off the Pacific crest trail in a mountain town called Mammoth Lakes. I walked back south on the trail after becoming violently ill and took a bus to Reno. I had already decided that I would be dropping out after a couple more hundred miles but this illness cemented my decision.

So now I must finish the sections that I have left- and I am back in mammoth now.

This plane:

20160619_115140-1600x900

This plane:

 

IMG_20160619_183914

And these free beers:

20160619_171826

Got me here.

As an added bonus I was able to meet up with Big Red whom I met during my 2014 hike. He is a really nice guy who has kept up with the hiking community much better than I have.

Now all I have to do is get supplied and take a bus to the trailhead and hopefully walk a few hundred more miles!

3 Days in Hawaii: The Big Island

I had the opportunity to take a longer weekend and visit Hawaii. This has been a bucket list item and I had a blast!

13217290_10154788574310130_592602764721193368_o

The hardest part for my planning was deciding what “visiting Hawaii” actually meant to me- and thankfully I had a travel buddy that helped broaden my scope of activities and enjoy the experience quite a bit more!

Overall- the plan was to get the most out of the least amount of time and I feel that we did a great job of hitting everything that we wanted- all while staying withing a decently strict budget. We flew into Kona and rented a car from Enterprise (I highly recommend spending a little more to go with a reputable car rental company). The overall thought process was to move about the island in a clockwise direction.

This resulted in a plan that included:

Friday night
  – grab car
  – Drive to Lookout – sleep in car
Saturday Morning
  – Waipio Valley Hike –early–
Saturday Day
  – Farmers Market Honokaa
  – Akaka Falls
Saturday evening
  – Pāhoa, HI
Saturday night
  – Issac Hale Park
  – Volcano National Park (night lava viewing)
Sunday Morning
  – Thurston Lava Tube
  – hike Kīlauea Iki Crater
Sunday Day
  – Volcano National Park
Sunday evening
  – Jagger museum and Volcano National Park
Sunday night
  – Snorkel with Manta Rays
Monday Morning
  – Hawaiian Bakery
  – Kahaluu Beach Park Snorkeling
Monday Day
  – Relaxing
  – Hawaiian food
While we missed the snorkeling at Kahaluu, we substituted a snorkeling adventure with Manta Rays! I can’t recommend this enough!
Sequence 01_18382ed
Overall I think that this itinerary fit our plans well and didn’t waste time at resorts or bars and we felt that we had a great vacation in the end.
Here are some more shots from the adventure:

This slideshow requires JavaScript.

As Promised: long delays between updates.

It may be evident that I have not been updating this blog very much as of late. Don’t worry- I have been having a ton of fun over the past few months, I have just been debating whether or not blogging hurts the natural world.

I have noticed a growing trend of trail trashers- those that are invited into the wilderness by social media to get “the perfect shot” or “tag” the trails with graffiti so they can leave their mark. I am reminded of what I saw in Southern California while hiking down there:

DSC06874

The issue with the perfect-shot people is that photos can end up trashing nature as people leave trails, cut bushes, and straight up modify nature to make it “prettier”. I do not see this as being helpful to the preservation of these resources for future generations. People who tag the trails are just big jerks- I got in an argument with one such person and they felt it was their right to put their marks on signs and trees. That argument ended in them informing me that I need to carry more paint remover as they were more empowered with their efforts. This didn’t make me happy, so I spent some time trying to figure out if my blog actually hurt nature.

I am still debating it- but I have deleted my photography page, and I am now not posting WTA trip reviews unless it has been a substantial time since the last- we don’t need 20 updates on the Rattlesnake trail each day.

So on to lighter news… here are some shots from the past couple months:

This slideshow requires JavaScript.

As is probably evident- I have been doing a lot of biking, mainly during my on-call rotation as I can explore near work and get some good exercise in (2016 totals so far: 540mi and 23,169 ft gain). My goal of this Winter was to avoid muscle loss due to inactivity- I am planning on going back to the PCT this Summer for a couple weeks and need as much muscle as is possible to be able to hop back into the higher mileage days needed.

Hope all of your worlds are spinning in the correct direction- and remember LNT when you are exploring. Here is a good reminder of what you can do to minimize impact: https://lnt.org/learn/online-awareness-course

Happy Trails, Friends!

The start of Oregon: PCT Section G

The completion of Washington left me feeling like my backyard had been explored. Further adventures on the PCT would now require more planning and coordination with rides and time off from work. My time on the trail in 2014 gave me the opportunity to meet some fantastic people- one of which was gracious enough to meet us super late in the evening to drive us to the beginning of Section G. Smitty- thank you very much!

The first night consisted of a 3 mile walk to a camp further away from people. There was a slight incline but it the mileage was easy. My pack was fully loaded with food and water but was only 16lbs- the campsite was about the halfway mark between the highway and Timberline lodge with lots of volcanic sand starting just after. I made it a few minutes in my tent before falling asleep. The drive down after work was enough to tire me out quite well.

12185749_10153070980071401_893081277_o

I had no idea that Timberline Lodge was located right off of the trail so when we saw it appear out of the fog and rain, we jumped at the opportunity to get some water and coffee. For those that have not been here before- Timberline Lodge offers an amazing buffet in the morning which hikers tend to feast on. The coffee was worth the $15 alone- I doubt that I will ever have coffee that tasted quite as good as they served. We were cold and fairly miserable as it was foggy, rainy, and windy so this boosted our spirits quite well.

12180843_10153070979906401_1069251439_o

(photo Credit goes to Jordan for this shot)

After spending 50min dreading the cold and enjoying the warmth- we continued onward. As the day progressed, it kept on raining harder and harder.

DSC07469edsml

There have been recent discoveries about this effect that water has on fingers- it appears to be evolutionary to help us grab onto things when immersed in water. In this case- it made my hands feel odd and I don’t think it was necessary for my trekking poles- but good on ya evolution… I guess.

Untitled_Panorama1sml

Besides this crossing and a couple other streams, this section was without many larger bodies of water. The sections in Washington all held plentiful lakes along the way but I guess this section could be good during mosquito season. Crossing this river took a little planning and wishful thinking. We made it safe and sound.

DSC07477edsml

After a while, we finally got a break in the rain. It was pretty and quite nice to have a break from the downpour. The weather forecast called it correctly when they said it would start to clear up in the late afternoon. We had hoped for this to happen as we did not want to camp in the rain. Hauling wet gear is not fun, sleeping in wet gear can be fatal.

We ended the day at 27.5mi- only 1.5mi short of my longest day on the trail. The light packs help a lot and so does the bike commuting that I have been doing recently. As winter approaches- we also were operating on a shorter daylight clock so it was getting dark as we setup camp.

DSC07489edsml

The following morning consisted of a lot of fog which collected on the trees and made it rain on us- it wasn’t a great way to start but it started to clear up as the day progressed.

DSC07495edsml

It was good to see familiar mountains from this opening- We could see Mt. St Helens, and Mt. Adams and Rainier off in the distance. We were mainly happy to be out of the fog. This is nearing the point where most PCTers hike down into the Eagle Creek area. After much debate we ended up missing the trail which cemented our plan to hike the official route (5mi longer too). I hope to come back and do a loop of Eagle Creek and another trail that connects two canyons- the Columbia River has amazing cliffs from the lava flows of the past which end up creating amazing waterfalls.

DSC07510edsml

Two miles later- we rounded a corner and got a full shot of Mount Hood- we hiked on the side of it the previous day but had no visibility which was a big bummer as we could only imagine how nice the views were.

Untitled_Panorama2sml

We finally got our first glimpse of the Columbia- a good sign that our hike was coming to a close. It was getting warm at this point and we had no water for ~8mi, any warmer and we would have had some issues with what water we had left.

DSC07562edsml

The trail resembled Fern Gully as we descended towards the river- found a salamander and this little guy.

DSC07574edsml

And a little further down the trail- we finished. With creaky knees, damp clothing, and a long drive ahead of us we quickly changed into clean clothes and drove North. We only did 25.5mi on the final day but the descent was a brutal one. 55mi in less than 45 hours with ~9kft elevation gain was a good amount to accomplish.

I did not ride my bike to work the next day.

Happy Trails everyone- looking forward to exploring more of Oregon soon!

Gothic Basin

DSC07388edsml

Gothic Basin is a fun and rewarding hike that follows an old mining trail and shares some of the route with the Monte Cristo trail. This area is full of geologic and mining history which are fascinating if you have a change to read into it.

DSC07413edsml

The rocks have fascinating features all along the way- but get a lot more interesting towards the top!

Untitled_Panorama2edsml

The primary destination of the hike is this lake, but there are a few tarns out there as well. I would highly recommend spending more than one day up here and summiting some of the awesome peaks in the area!

DSC07460edsml

I made sure to bring my MP3 player and listen to gothic music just for a few chuckles. It was a good spot for some good music!