With the responsibility to support DevOps needs for 2000+ engineers in the Data center and Cloud networking team, the existing infrastructure was a bottleneck to support the requirements of complex engineering projects. These requirements include a mix of mono and multi-repositories, a rapidly increasing number of microservices, and deployments across multiple clouds in addition to on-prem deliverables. The need for a fast, secure, scalable, portable, and efficient infrastructure applicable for all the products was clear, and it had to be based on open source (OSS) to be cost-effective while delivering an excellent developer experience.

The Accordion framework has evolved over the past three years as a comprehensive CICD solution embracing Containers & Kubernetes ecosystem for any SaaS or on-prem project at Cisco, offering huge productivity boosts to thousands of engineers and multi-million dollar savings annually. The Accordion framework is optimized for Cisco environments with many innovative features unmatched by commercial offerings while adopting industry-leading best practices. The extensible design and open architecture allows the Accordion framework and tools to be reused and adapted for applications beyond CICD across Cisco product/service offerings.

Architecture

Features

1) Accordion Pipelines/Workflows is a truly declarative and low-code cloud-native solution that simplifies the implementation of pipelines with agility. Developers can build complex workflows easily with reusable, extensible templates that support high-level abstractions in simple declarative YAML. Accordions can expedite development by eliminating the necessity to master domain-specific languages like Groovy that are difficult to manage at scale. Accordion encapsulates the underlying complexity of the infrastructure and integration with open-source components, using templates and plugins written in Golang.  

As a simple use case, developers can trigger customized workflows by assigning labels to PRs in GitHub; internally, dynamic Kubernetes agents implement these dynamic workflows in multiple stages that securely run one or more containers attached to storage volumes. The integrated observability for fine-grained resource consumption at every stage enables quick troubleshooting to achieve higher SLOs.

2) Accordion Snapshots work with any cloud-native block storage to support dynamic snapshots for reusable pipelines and enable up to 5x faster incremental builds regardless of build environment/language, hiding the internal complexity from the user. It also enables unique features of fast incremental builds on Kubernetes cluster. Pipeline snapshots feature simplified restart from failure scenarios, thus saving lots of time.

3) Accordion GitOps extends GitOps paradigm beyond the traditional approach, which focuses on reconciling Kubernetes cluster states to deployment configurations in Git. Accordion GitOps provides logical abstractions for tracking deployment across Kubernetes clusters in multiple cloud environments (e.g., Dev/QA/Stage/Prod) and integration with bug tracking systems for end-to-end SDLC. The implementation provides flexibility to support multiple deployment tools and cloud providers. In addition, this framework enables GitOps innovations to simplify test automation for feature branch development workflows.

4) Observability and Analytics stack provides plug-and-play functionality for monitoring the CI/CD platform, setting up alerts, and visualizing the quality metrics. Real-time Kubernetes agent monitoring and pipeline crash analysis provide an added advantage in self-monitoring pipelines. Below are a few of many dashboards from Accordion observability and analytics stack.

Results 

1. Cloud native transformation of DevOps with huge cost savings. In the past three years, the Accordion project and its innovations have unleashed significant productivity gains and multi-million dollar cost savings. It has transformed the DevOps experience with high-quality CICD automation delivered quickly, empowering developer agility with a self-service model. What used to take weeks is made possible in days or less, sometimes 90% faster than the traditional approach. Fast-paced SaaS projects would not have been possible without Accordion, specifically due to limitations in infrastructure and people resources caused by the pandemic.

2. Build time improvement resulting in increase of release velocity. Innovative use of open source with distributed caching helped reduce the build times, enabling a faster iterative development process. Accordion snapshot builds are up to 5x faster.  

3. Higher quality with improvement in operational efficiency. The Accordion infrastructure has exceedingly small downtime with MTTR in minutes. The built-in observability of Accordion helps predict operational demands more accurately and sends real-time alerts with relevant debugging info to help maintain high SLO. The self-service model for developers with Webex Bot alerts helps reduce Opex requirements for DevOps. The Accordion architecture is highly portable across platforms and has been tested on Google Anthos (Cisco IT) and AWS EKS. 

Conclusion

Accordion has emerged as a comprehensive cloud-native CICD solution based on OSS with many innovative features unmatched by commercial vendors. It is being used by 1000+ engineers for multiple products, delivering over 16,000 builds per month with an availability SLO of 99.999%. Its dynamic runners based on incoming pipeline requirements with declarative manifests, dynamic workflows, and pipeline snapshots features save infra costs by manifolds.

Acknowledgments

We want to express our gratitude to the Data center and Cloud networking DevOps team for their valuable contributions to this project.

References

Jenkins Pipelines

Kubernetes based Jenkins Pipelines