In the dynamic realm of cruise line marketing, implementing efficient automation processes is paramount. This case study explores the journey of Norwegian Cruise Line Holdings Ltd. Cruise Line's (NCL) adoption of CI/CD practices for SAS Viya platform to streamline administrative tasks. NCL achieved significant reductions in deployment time, improved durability, and enhanced campaign management efficiency. Delve into this study to uncover how NCL navigated challenges, realized key benefits, and paved the way for future enhancements in their CI/CD journey.

Principal and its features in regards to CI/CD processes.

Norwegian Cruise Line (NCL) is one of the world’s largest cruise lines headquartered in Miami.

NCL extensively utilizes technology to deliver flexible and customized cruising options for an enhanced passenger experience, e.g.:

• Key card contactless payments reducing queues.  
• Guests have access to a robust mobile app and interactive portals for planning personalized schedules with dining/event reservations.
• Virtual queues and paperless embarkation enable a smooth onboard experience.  
• Cutting-edge in-room TVs, widespread WIFI, and touchscreens ease access to information and amenities.  
• Self-service portals provide on-demand booking abilities.  
• And many more.

By leveraging these customer-facing and operational IT systems ranging from IoT to mobile devices, NCL vessels empower guests with more customizable cruises to provide a best-in-class user experience.

What sets the cruise line industry apart is its fleet, presenting distinct technological challenges. From a tech perspective, these challenges include:

• Operating in a Multi-Production-Environment Setup. Multiple vessels in a fleet lead to multiple production environments maintenance in concurrent mode, introducing complexities in orchestration and coordination.
• Dynamic Timezone Shifts. Frequent shifts among different timezones within each environment add an extra layer of complexity, requiring processes that will seamlessly adapt to timezone fluctuations.
• Intermittent Connectivity Issues. Environments intermittently offline or grappling with poor connectivity present a further layer of intricacy, demanding resilient processes capable of accommodating varying connectivity conditions.
• Diverse IT systems on vessels. The deployment of different IT systems on vessels, tailored to unique features for each vessel, adds complexity to maintaining a cohesive technological infrastructure.

These challenges make it difficult to meet modern requirements for maintenance, fast change implementation, and overall durability. NCL, in alignment with industry standards, encounters and addresses these challenges, showcasing a commitment to technological resilience and adaptability in the face of the unique operational constraints inherent to the cruise line sector.

Problem and Solution

Initial State and Encountered Issues

In the pursuit of delivering an exceptional customer experience, Norwegian Cruise Line (NCL) strategically opted for the SAS Viya platform. This versatile solution plays a crucial role in analytics, marketing campaign execution, and retrieving customer insights. Notably, the aspect of marketing campaign execution holds paramount significance being a front-facing system. Thus it directly impacts customer experiences during cruises.  

In adherence to standard practice, the commencement of new systems and processes involved the execution of Proof of Concept (PoC) and/or Minimum Viable Product (MVP) phases to ensure feasibility from a business standpoint. Only upon project justification did we embark on planning the IT landscape, which naturally included considerations for implementing CI/CD. The expected increase in processes and features, along with the large number of marketing campaigns and vessels, made CI/CD implementation inevitable.

The initial releases, while marking a significant milestone, were not without their challenges:

1. Time-Consuming Manual Processes: Tedious manual build, testing, and deployment processes that significantly slow down release cycles. This hinders the ability to deliver features and fixes rapidly to meet market demands.
2. Lack of Visibility and Traceability: Tracking build statuses, code changes, and deployment history becomes challenging without automation. This makes it difficult to quickly pin errors back to root causes.
3. High-Risk Deployments: Urgent releases underscored gaps in our deployment process, amplifying associated risks. The manual nature of the process not only left room for human errors but also lacked crucial rollback and recovery mechanisms, heightening the stakes during critical deployments.
4. Inefficient Defect Remediation: Identifying, fixing, and retesting bugs in the application lifecycle requires significant rework without CI/CD.

Those issues are exacerbated by cruise lines features mentioned earlier, i.e.:

• Operating in a multi-production-environment setup,
• Dynamic Timezone Shifts,
• Intermittent Connectivity issues,
• Diverse IT systems.  

In summary, we identified initial complexities that underscored the critical need for a robust CI/CD framework.

What CI/CD is and why it is important?

A quick recap of the CI/CD concept.  

Continuous integration and continuous delivery (CI/CD) are vital for any modern company utilizing IT systems, for several key reasons:

1. Faster time-to-market: CI/CD automation allows teams to release new features, updates, and fixes in an accelerated manner, taking days or weeks instead of months. This acceleration ensures a prompt delivery of value to customers.
2. Improved software quality: By automatically integrating and testing code frequently through a CI/CD pipeline, defects can be caught early. This results in more robust software that enables mitigating potential outages or issues after production deployment.  
3. Lower failure rates of releases: CI/CD, with its comprehensive test automation and incremental code deployments, reduces the batch size of changes. This minimizes rollout risks and facilitates quicker recovery in case of problems.
4. Better scalability and flexibility, especially in conditions of rapid growth of the number of environments and / or objects to deploy.  

CI/CD principles empower technology and product teams to work at maximum efficiency, enabling businesses to swiftly achieve strategic goals in response to the market's demand for rapid innovation. Ignoring the benefits of CI/CD is not an option for modern organizations heavily reliant on IT. It represents a crucial advancement towards organizational maturity in the IT domain.

Objectives of CI/CD implementation

The objectives of CI/CD implementation fall into several categories aimed at enhancing various facets of the software development lifecycle:

Release velocity

• Minimize the time needed for a deployment

• It should take minutes in regular cases
• Could be extended up to a 30-minute interval maximum in case of poor connection

• Consider potential time lags and network outages, and optimize the deployment process accordingly

Durability

• Ensure the capability for rollback or restoration post-release deployment if required
• Adjust scheduling via CI/CD, including an option to turn off processes

The overarching strategic objective is to provide the system with scalability while preserving maintainability and durability.

Implementation

We identified several key areas that should provide a robust CI/CD process:

• DB migrations
• Configuration maintenance
• Code compilation
• Artifacts maintenance
• Release management

The best-in-class tools were employed at each area. All the tools selected were carefully harmonized to operate seamlessly within a unified process, accessible through a single interface. This integration ensures a convenient and reliable toolset for administrators, facilitating efficient management of the CI/CD pipeline.

Results and Impact

Outcomes

Release velocity:

• The full redeployment takes up to 5 minutes for each vessel. In case of poor connection overall time extends up to 30 minutes.
• Deployment time to a new vessel from scratch has been reduced significantly (by 90%).

• Time lags and network outages impacted hugely on manual deployment due switching context and waiting for the feedback during CLI and GUI interaction. The automated process handles it easier.

• New campaigns’ time-to-market has plummeted by 80%.

Durability:

• Median time 1 hour for a non-critical issue - to revert to the previous stable release
• Median time 3 min for a critical issue - to stop outbound communications

Deployment time impact

From the second vessel deployment onwards, all functionalities were executed through CI/CD pipelines, enabling rapid onboarding of new vessels within minutes.

A significant workflow enhancement was the streamlined DB schema migration process, ensuring no loss of DB views, indexes, or primary keys due to human error in the automated deployment process.

Moreover, the ability to deploy code swiftly became essential for integration testing, allowing automatic installation of test branch code and execution of tests without impacting the main deployment.

Over an 8-month period of utilizing automated deployment, the code was deployed approximately 750 times. Manual deployment time varies based on the object type, with an average vessel deployment estimated at around 1 hour due to the use of a Terminal Server on the vessel, resulting in significant input lag for each user action.

Based on average time difference we can conclude that approximately 680 hours of team work were saved through automated deployment within those 8 months.  

On average, there were around 0.4 deployments per day. However, deployments were not evenly distributed over time. During active deployment and testing phases, more than 10 deployments could be completed in a single day.

Manual execution of this volume of deployments would be challenging and lead to a slower testing cycle. Thus, the automated deployment process resulted in even greater time savings.

Key benefits

The key benefits from adopting CI/CD:

• From a business perspective:

• Faster time-to-market
• Increased customer satisfaction

• From a technical perspective:

• Faster testing → less time for testing while improving its quality
• Scalability and flexibility
• Quick rollback and recovery

Hands-on experience

The key campaign management specialist on the project, Malika Kakhirmanova Kakhirmanova, sharing her thoughts:

“We had CI/CD process ready right after the deployment on the second ship, and we were able to evaluate its efficiency when we piloted next ships. Even easy operations take a lot of time when you need to switch between a large number of environments with intermittent connectivity. Now the process goes smooth and fast, and what is more important - there are no human errors, such as forgotten steps from the Deployment Guide or incorrectly substituted variables. As a result - we can shortly extend our constantly growing process to more environments, without any pain expected.

The process of development of new marketing campaigns didn't change, however, their testing did. We don't test every step on each ship, because we know that technically everything is the same, we just evaluate the result and monitor the process.

For my role, I have more time for more technical/business and not-so routine tasks, which encourages me”.

Business leader and owner of the project, Ethan R. Rasti, expressing his perspective:

Using a central CI/CD platform is one of the main components of our DevOps efforts. We currently have more than 30 ships, and having such robust CI/CD process has enabled us to easily manage changes fleet wide and also monitor the status of each job on every ship in a central command center. Before this solution, we had to login to every ship in order to push a change. This effort has been saving us time and resource”.

In essence, the implementation of CI/CD processes has resulted in a transformative impact, enabling to operate at peak efficiency, leveraging SAS Viya platform to enhance customer experience and make it outstanding by providing means to scale fast, navigate the dynamic requirements, and address the distinctive challenges posed by the cruise line industry.

Recommendations for others implementing CI/CD processes

After the journey of CI/CD implementation, we have distilled key recommendations built upon gained experience and insights:

Start Small and Incrementally Scale

• It’s not necessary to employ CI/CD tools for the system you just releasing. View CI/CD as a part of the infrastructure. It should be implemented only after you are confident the tool you are adding is needed. Begin with proof of concept (PoC) and Minimum Viable Product (MVP) projects to validate feasibility and benefits. Fortunately, CI/CD processes can be implemented gradually within an already operational system.

Keep the big picture in mind and address challenges systematically

• The big picture aids in distinguishing essential requirements from optional ones. In our case, unique features in the cruise line industry, such as multiple production environments and dynamic timezone shifts, were identified as essential. Systematically tackling these challenges ensured the development of a resilient and adaptive CI/CD framework. Anticipate challenges, plan contingencies, and keep the implementation on track.

Treat CI/CD Implementation as a Project

• Recognize CI/CD implementation as a project. Dedicate time to planning, set realistic expectations, and communicate them organization-wide. This proactive approach manages stakeholders' expectations and ensures a more predictable and successful implementation.

Choose tech stack wisely

• Given the multitude of tools available for CI/CD processes, decide wisely whether to invest time in thorough research or not. We decided in almost no time what the tech stack should be used based on the team’s experience and positive references. In hindsight we agree the decision was right: given all the complications we faced, wasting time on it wasn’t worth it.

CI/CD Requires Cross-Functional Collaboration

• Acknowledge that CI/CD implementation is not just a technical endeavor but a holistic development involving various teams. Ensure active involvement from planning, testing, and other relevant departments to foster cross-functional collaboration and alignment with organizational goals.

Future enhancements and next steps

CI/CD represents just one facet of the comprehensive infrastructure that a mature system should encompass. Among others, it includes monitoring, alerting, auto testing, log collection, and high-availability setups.

Currently, we are actively engaged in the implementation of monitoring and alerting processes. Please stay tuned, as we will certainly keep you posted on our progress.

‍

‍