Platform modernization and engineering for large organization
Snapshot
Our client, a global large-scale non-profit organization, has a charter to improve healthcare and end poverty around the world, while focusing domestically on expanding education opportunities and access to information technology. The organization faced an aging IT platform with software containing copious lines of custom code housed within physical servers. There hadn’t been an upgrade of the organization’s enterprise software in almost two years, and the expiration of hardware and Microsoft extended support was fast approaching. Updating their enterprise software system was imperative to the future of the business. To reduce IT support and hardware costs, the organization would move their system to virtual servers
The organization’s employees are challenged to reward grants to a significant number of worthy causes within the fiscal year. For the organization to meet its goals, employees would require continuous access to SharePoint platforms that hosted sites for teams and departments such as IT, security, the legal review and processing systems, and the grant application and review systems. The Kalles Group was engaged to join Infrastructure and Cloud Enablement (ICE), the team that supports nearly all IT platforms which in return, support the entire organization and the business worldwide. To ensure unobstructed service to employees, and make smart use of internal resources, timing and tight coordination of this project would be critical.
Challenge
Timing challenges required careful pre-planning and scheduling across multiple teams. Each month, all QA resources were consumed with testing production after OS patches were applied, so any changes were limited to a two-week window of QA availability. Also, some external teams had to approve of downtime because of critical tasks that required service availability.
While production was the number one focus, the pre-production environments were just as critical due to the number of development teams working on the next feature and release to production. Because of delays in resolving production server issues, Kalles Group team members were tasked with migrating all staging server migrations before production could be approved.
Approach
In initiating the project, evaluating the health of the environments uncovered missing Installer Cache files. SharePoint validates the Installer Cache files in order to determine that the cumulative upgrade can be applied to each environment. All missing files would need to be located and put in place.
Working on the staging servers and production servers separately goes against best practices that dictate that when a change is completed in staging, that change should be updated in production immediately in order to ensure all lessons learned in staging are still fresh and the process is still within memory
With these timing and task requirements, to ensure the cumulative upgrade would happen according to plan the Kalles Group team members completed the task of meticulously documenting decisions and surrounding logic for changes to the staging servers so they could be applied in production when those servers were migrated.
While migrating a database at a point where a rollback was not possible, an SSRS problem was identified. The Kalles Group quickly identified a solution and after a fast and clear explanation of the issue, the problem was resolved and additional downtime was avoided for a deployment that was already scheduled to go deep into the night.
All this work was affected by limited internal QA resources, server space, and bandwidth resources. This meant it was critical that deployment plans were communicated clearly ahead of time with QA teams, so that changes could be made when server and bandwidth resources were available. There were times when server drives ran dangerously low on space, which risked bringing the process to a halt. The Kalles Group team members carefully managed the process with a documented plan that would ensure first-time problems encountered in migrating server environments would be avoided in the migrations of the other servers.
Results
Once the previously released security patches which were installed, the platform was now up to Microsoft’s compliance and supportability requirements so the company had access to Microsoft software support. With the server migration complete, server support was no longer limited to a single IT employee who was required to physically be in front of the server. Servers can now be monitored via the Virtual Machine UI, which means the ability to create virtual environments is more broadly shared across IT. Off-shore teams can directly access the server resources as needed, without having to funnel requests through a single employee
Savings to the client were significant. The support and hardware costs for maintaining the previous systems were significantly reduced and could easily be forecasted into the organization’s financials. Also, the risk of hardware failure was remedied just in time before the warranty of the physical servers would expire at month-end.