How do you deploy a new remote site?

Iristheangel

I'm curious to see what other network engineers think about my methodology. I'm in the middle of a huge network refresh and adding Wi-Fi/CCTV to about 21 sites. Part of this involves configuring new routers and about a dozen or so switches for each site as well as utilizing existing old switches that are onsite (note: legacy equipment usually doesn't have SSH or update code) and having our wiring vendor who is pulling all the fiber and wiring for these sites physically install the equipment and turn it up for me. Usually these vendors don't have laptops on them so if something goes wrong with my config, It gets messy.

The way I've been doing it has been to configure the router and switches with pretty barebones configurations: Routing, SSH and Telnet, VTP Transparent mode, VLANs, port security local login, etc. Once I get the site up and running, I go back and update the code on legacy equipment, add DHCP snooping, get rid of telnet completely, add TACACs, VTP server/client, change all the trunks to have a different native VLAN (blackhole VLAN), add port descriptions, etc and lean on the "reload in 20" command. After I'm done with this, I create a logical and physical diagram, map the IDF/MDF locations on floorplans, copy the final configs, create a Sitepak with everything about the site and move onto the next.

I've had a pretty high success rate with little to no interruptions using the above methodology but I've been a network engineer for a little over a year now. My new co-worker has been doing this for 8+ years at other companies and he's a bit different. He adds EVERYTHING all at once including TACACS, route-maps, DHCP snooping, etc to start and sends it out. I haven't seen him bring up a site yet without issues due to his configs but of course, he's pretty insistent about sending the routers/switches out with the final config to start.

If I just took it as anecdotal, I would say based on my record of successes compared to his with deploying sites, I'd be right here but anecdotal evidence isn't always the best way to go. What do fellow network engineers say? What methodology do you employ when deploying a new site remote with dumb remote hands or mixed new/legacy equipment?

Thanks for reading!

shodown

Depends on the buget.


Usually when I deploy a site.

1. Site suvery. What services do they need, alarms, door buzzers video. What services do they have in place.

2. I get copies of the existing devices configuration and find out what is needed and whats not.

3. I put baseline configs on the new equipment, IP's, Routing,QOS, VLANS. No ACL or traffic blocking configuration.

4. Deploy the site and add the ACLs, policy maps, ETC. Start a test plan based on the information I got in Step 1.

If there is any legacy equipment that plays a major piece like a core switch or something I also start fresh with that gear as well treating it as its new.

W Stewart

Not really on the networking side of things but I just have to say. Doing it for 8 years doesn't always mean doing it the right way for 8 years.

paul78

As in many things IT, I don't think there's necessarily a one-size fits all approach. While I don't do hands-on network engineering, I'm a consumer of the services that it being provided for my line-of-business. So at the company that I work, the expectation is that the network engineering teams would take their direction from the businesses.

In the scenario, that you outlined, my business requirements would have mandated an approach that would be based on the critical nature of the sites. For some sites, I would not allow an external cable-vendor to rack/stack the network equipment so I would expect that the network equipment be pre-configured to the point where local support can just rack it.

For sites that are more critical in nature, I would fund travel for the network engineer to be onsite during the upgrade. But in order to reduce downtime (usually less than 15 minutes downtime during the maintenance window), I may require that the final configs be pre-built and tested before it is deployed.

I think that either of the approaches that you mentioned are probably fine. My concern with your approach is that it implies that the configs may not have been pre-tested.

Ultimately, in our environment, I would provide the requirements but I leave it to the network engineers to do their job. But my requirements always include some level of pre-deployment testing and a roll-back plan.

Iristheangel

Unfortunately in our environment, we don't have the luxury to travel to every site during an upgrade and our remote hands don't always pick up everything in the physical topology. Configs are something I understand and it's not a matter of me not testing them out but more of that you never know what you might be surprised with. I.e. I had a network refresh for a site that had about 18 IDFs where random switches were daisy chained together and there were mixed vendor switches: old Cisco 2950s mixed with craptastic D-Link switches and even some hubs. We were only replacing the switches with APs plugged into them with 2960' so they could get PoE. When one of the switches were plugged in, the trunk started flapping. I could see the other switch on CDP neighbors and the ports were configured with the most basic of port configurations: "Switchport mode trunk"

It turned out that we had a "surprise" D-Link piece-of-shite in the way that was interfering with trunking and causing flaps. As soon as that was replaced with one of the old 2950s, BAM. Came right up and everything was fine.

If I have configured VTP on both switches prior to mailing the out, used the trusty "macro apply cisco-switch $native_vlan 999" (my blackhole vlan of choice!), enabled DHCP snooping on the switches, etc, I would have having to troubleshoot all those things first and start narrowing down what the issue could be before finally finding out that it was something completely out of my control.

That's why I use my method. I keep it simple at first because there are always the possibility of unknown factors popping up. Especially when you are working on something remotely.

Believe me, when I configure my site equipment, I hook it all up EXACTLY based on what I know the physical topology to be in our lab and configure from there. I label each port and switch with a label maker for all the uplinks to other switches or router. I include notes on which switch is replacing what switch and the location. I'm to the point of being OCD about it and the followup documentation when I'm done with a deployment.

As far as customer needs and requirements, that's a given. My original post is more about the methodology employed when remotely bringing up a site and how you do configurations.

paul78

Sounds like you business requirements has a few interesting constraints.

In your situation, your colleague's approach of using a "sunny day scenario" deployment method seems a bit over-optimistic.

Generally speaking I would never advocate a "sunny day" deployment if changes cannot be pre-tested and confirmed based on an understanding of the existing environment. If such conditions exists and I am not willing to fund a site-survey, my expectation would then be a deployment method that reduces work-effort expenses and overall risk.

A more flexible approach which your proposed would likely be more appealing to me. A fully-preconfigured approach seems like an inefficient use of work-effort. And at worst, could introduce additional risk to the environment if there were assumptions made about the site's network which may not be accurate.

Edit - Bear in mind - my comments are not from a network engineering best-practice perspective. It's from the perspective of technology management for a line-of-business that would be ultimately responsible for a site's uptime and business processes.

Dieg0M

Depends of the client. I have done both but I usually tend to lean in the bare bone configuration way because client requirements often change. It also depends how much time I get for the planning phase. If the client needs it implemented really quickly I often just slap bare bone configuration and, while the physical infrastructure is being prepared, I create the configurations. As soon as the physical links are up I remote in every device and make the appropriate changes.

darkerz

I automated my efforts for Access & Distro switches. Quite honestly, I cannot believe I used to do anything manually.

Only thing I enable is for the switch to receive a DHCP address and telnes access w/ a local acc.


...

1. I ship it.
2. I have it plugged in and powered on, on a specific port.
3. Run script.
4. Finds new device
5. Finds templates & configs for that subnet (This allows us to specify unique, specific configs in a central DB)
6. Config happens, pulls all show outputs and parses it into a neat document.
7. Finalizes configurations, SSH, TACACS+, etc.
8. Wr mem, reboots, script will save the log of every change, success, failure, show output, etc. in a detailed folder with a log file of commands put in, checked, etc. and another file that checks sh runs and parses it so we know by title what is the name, device type, license, etc. of the device
9. Configured

What I experience;

1. Vendor ships it
2. Switch is plugged in. Before hand, I tick a few boxes on my program to what I "want it to be"
3. I press my run button.
4.Switch comes up and it works 1 to 4 minutes later, depending on type of switch.


I'm working on the same thing for our Routers, Firewalls and AP's as well.


I love scripting & creating GUI's for my work

phoeneous

darkerz wrote: »

3. I press my run button.

What program?

ptilsen

I am not a network engineer, but in the past when I've deployed equipments to sites remotely, I configured anything that wouldn't conceivably cause an issue with connectivity before hand. Everything else would wait until the equipment is onsite, connected, and there's connectivity. I wouldn't want to guess what best practice is, but I will say I doubt there's going to be a "right" answer. It's going to vary a lot based on the specifics of the situation.

I will echo W Stewart. While more experienced engineers should certainly not be disregarded, that they have more experience is not necessarily going to mean they're actually doing things better or have more skill. I would say it's weakly correlated at best. I've worked with people who've been in the industry longer than I've been alive who were totally incompetent and had almost no technical knowledge. That is far from the norm, but it's important to remember that experience does not equal skill.

Hopefully more people who actually do the type of work you do day-to-day will weigh in.

DevilWAH

I put on a config I know will allow me to connect and I have tested. So how much config I put on really comes down to how much time I have had to test.

So I have done the full config as it will run in production, sent this to the supplier and got them to installed it, had the device delivered direct to the remote site and had a junior engineer plug it in and its up and running with out issues. I have also when time has been tighter, just set up the very basic config that will allow me to remotely connect and done the entire config remotely.

Generally for edge devices at remote sites I do all the config first, while for access and internal devices I do less upfront and more remotely. I have started now I have CISCO Prime creating a lot of baseline configs that I can push to devices which makes setting up devices remotely a lot easier.

Out of interest rather than using just "reload in..." you can try this

https://supportforums.cisco.com/thread/2237572

using an IP SLA monitor to trigger a reload, so if you screw it up (the technical term ) 60 seconds in, you don't have to wait 19 min for the timer to expire before you are able to carry on. I normally set the time out at about 60 seconds and I still use the "reload in ..." as a back up.

fredrikjj

Where I worked we generated configs with scripts and fully configured each switch before shipping it. The the input for the script was basically a simple database containing information on which networks and vlans should belong to each site. There was also some layer 2 redundancy stuff that was specific to each site. The alternative would have been to have a template and then manually change the right variables in a text editor before configuring the device, something I can see leading to human errors.

Shipping in this case was just a matter of driving 5-15 minutes since most sites were within the same city, but things never broke due to bad initial configs as far as I can remember. It also should be said that the entire enivornment was very uniform, with all ~100 sites having more or less identical topologies and gear, with the number of access switches being the only thing that changed.

Iristheangel wrote: »

...even some hubs.