In addition, the team plans to adopt Amazon OpenSearch Service’s fine-grained access control, which enables different roles to access data at the index, document, and field levels. “We’re really careful with our proprietary information and private user data, so we set up a whole pipeline with that in mind,” says Zhu.
Security was also an important factor for Pinterest’s observability team as it was considering the migration to Amazon OpenSearch Service. “And since we only have one person managing the pipeline, that shows immense resource efficiency.” With only one engineer overseeing its Elasticsearch deployment, the team can turn away from low-value work to focus on innovation and other business-critical tasks, such as exploring new use cases and adding more resources to the log search. “Using, we can see that the scale has grown tremendously,” says Zhu.
PINTEREST DAILY ACTIVE USERS SOFTWARE
Currently, Pinterest is able to monitor and issue alerts for 20 new software deployments a day. By the end of 2020, the team expects to be able to ingest over 3 TB of data per day.
In just 1 year after the migration to Amazon OpenSearch Service, Pinterest’s observability team went from ingesting 500 GB of data per day to 1.7 TB per day. “And Kibana is open source, which makes using the tool much, much easier-if we have questions about using it, we can find the answers online ourselves.” Overall, Amazon OpenSearch Service enables Pinterest to quickly find and resolve issues as part of its continuous integration and deployment of software, helping Pinterest ship new features to Pinterest users faster. “The cost is much less because we don’t spend resources to manage the infrastructure,” says Zhu.
PINTEREST DAILY ACTIVE USERS FREE
Also, Amazon OpenSearch Service’s support for Kibana-a free and open user interface that lets users visualize Elasticsearch data-made it more readily accessible for Pinterest engineers. The move made sense: Pinterest was already using a variety of AWS services to scale its processing, storage, and data analysis workloads, and Zhu’s team determined that the move would reduce costs. In the second half of 2019, with the third-party vendor’s contract expiration approaching, Zhu’s team began migrating its data to Amazon OpenSearch Service. Pinterest signed a 3-year contract with a third-party vendor, but that arrangement presented new challenges: the vendor’s high licensing cost drove up the overall cost of the solution, and its unique query language produced a steep learning curve for Pinterest’s software engineers. But the costly operational overhead and the ever-increasing volume of daily data necessitated a change. Initially, Zhu’s team was self-managing open-source Elasticsearch software to monitor log data and troubleshoot issues with software deployment. Pinterest’s search for a cost-effective, scalable solution for log analysis began in 2016. “We use metrics to detect all incidents when they happen, but log search is the main tool we use to find out what’s causing them,” says Wei Zhu, a staff engineer for Pinterest’s observability team. With Elasticsearch, the team can identify problems during software deployment, then quickly analyze logs to troubleshoot root causes.
The Pinterest observability team relies on Elasticsearch to monitor and issue alerts for new software deployments on the main Pinterest site. There, Pinterest was able not only to scale its log analysis capabilities but also to reduce operational burdens on its software engineers, improve the security of its proprietary information and private user data, and save costs by as much as 30 percent. Pinterest moved to a third-party proprietary Elasticsearch solution but ultimately found the cost to be unsustainable and the solution to be unable to scale with demand.įaced with the enormous demand for faster, more efficient log analytics at a lower cost, Pinterest moved to managed analytics using Amazon OpenSearch Service on Amazon Web Services (AWS). It required constant administrative work from the company’s engineering staff, resulting in ineffective data analysis and an unsustainable operational overhead. But the self-managed open-source Elasticsearch tool the company used to search and analyze the data couldn’t handle the scale. In 2016, Pinterest-one of the largest visual-bookmarking tools and social networks in the world, now with 400 million monthly active users and growing-was creating 300 GB of logs daily, and that volume was increasing rapidly.
0 kommentar(er)
