Elasticsearch is a popular open-source search and analytics engine that allows users to store, search, and analyze large volumes of data in near real-time. Amazon Web Services (AWS) makes it possible to deploy a managed Elasticsearch cluster that is easy to deploy, scale, and manage.
With Elasticsearch on AWS, users can take advantage of AWS’s global infrastructure to deploy Elasticsearch clusters in regions around the world. This makes it easy to ensure low-latency access to data, even for applications and services that are deployed in different regions.
Elasticsearch on AWS can also make it easier to manage and monitor Elasticsearch clusters. Depending on the cloud service used to deploy Elasticsearch, users can get access to features like automatic scaling, automated backups, and integrated security controls.
In this article:
- Deploying Elasticsearch Services on AWS
- Getting Started With Amazon OpenSearch Service
- Best Practices for Running Elasticsearch on AWS
Deploying Elasticsearch Services on AWS
Amazon OpenSearch Service
This is a managed search and analytics service by Amazon Web Services (AWS). It is based on the open-source Elasticsearch project, and provides a scalable, reliable, and fully managed search and analytics solution for applications running on AWS.
It lets users set up and run Elasticsearch clusters without having to worry about the underlying infrastructure, configuration, or maintenance. It is fully compatible with the Elasticsearch API, which helps to easily migrate applications that use Elasticsearch to OpenSearch Service.
Features and use cases
OpenSearch Service provides various features and capabilities, including full-text search, real-time analytics, geospatial search, and security and compliance controls. It integrates with other AWS services, such as Amazon S3, Amazon CloudWatch, and AWS Identity and Access Management (IAM), and provides add-ons and plugins to enhance functionality.
Amazon OpenSearch Service is suitable for a wide range of use cases, including log analytics, eCommerce search, security analytics, and business intelligence (BI). It is available in multiple regions and offers pay-as-you-go pricing that allows users to only pay for what they use.
Amazon OpenSearch Serverless
This serverless version of Amazon OpenSearch Service automatically scales up and down based on the workload’s needs. There is no need to provision, manage, or maintain any infrastructure. It integrates seamlessly with other AWS services, including AWS Lambda, AWS Glue, and Amazon S3.
Amazon OpenSearch Serverless uses a feature called UltraWarm, which is an advanced data tiering technology that helps store and analyze large amounts of data cost-effectively. UltraWarm enables users to store and analyze data that is infrequently accessed, with up to 900% more storage capacity at a much lower cost than traditional hot storage options.
Features and use cases
Amazon OpenSearch Service Serverless provides the same search and analytics capabilities as the regular OpenSearch Service, including full-text search, real-time analytics, and security and compliance controls. Users can also use Kibana for data visualization and analysis.
This service is suitable for use cases that require ad hoc or exploratory analytics, data archiving, and batch processing. It is also suitable for building serverless applications that require search and analytics capabilities.
Getting Started With Amazon OpenSearch Service
Getting started with Amazon OpenSearch Service is relatively straightforward. Here are the basic steps you need to follow to get started:
- Sign up for an AWS account: To use Amazon OpenSearch Service, you need to have an AWS account. If you don’t already have one, you can sign up for an account on the AWS website.
- Launch an OpenSearch cluster: To launch an OpenSearch cluster, you need to navigate to the Amazon OpenSearch Service console in the AWS Management Console. From there, you can create a new domain by selecting “Create a new domain” and specifying the cluster name, instance type, and other configuration options.
- Configure the cluster: After you launch your OpenSearch cluster, you need to configure it to meet your needs. This can include setting up security controls, configuring access policies, and configuring other settings based on your requirements.
- Index data: Once your OpenSearch cluster is up and running, you can start indexing data. This can include loading data into the cluster using the OpenSearch API or using tools like Logstash to collect data from external sources.
- Search and analyze data: Once data is indexed, you can use the OpenSearch API or Kibana to search and analyze the data. With Kibana, you can create custom visualizations, dashboards, and reports based on your data.
Best Practices for Running Elasticsearch on AWS
Running Elasticsearch on AWS requires careful planning and implementation to ensure that the search and analytics engine is optimized for performance, scalability, and cost-efficiency. Here are some best practices to consider when deploying Amazon OpenSearch Service:
Dedicated Master Nodes
Dedicated master nodes are configured to handle cluster management tasks, such as managing node membership, creating and updating indices, and handling failover. These tasks are critical for the stability and reliability of the cluster, and are separate from data indexing and search tasks.
By default, Amazon OpenSearch Service clusters have three data nodes that also serve as master-eligible nodes. However, for larger or more complex clusters, it is recommended to use dedicated master nodes to ensure cluster stability and reduce the risk of data loss.
How dedicated master nodes work
Dedicated master nodes are separate nodes that are optimized for cluster management tasks. These nodes do not store data, but instead coordinate the cluster operations and handle the metadata. By separating the cluster management tasks from data indexing and search tasks, the dedicated master nodes reduce the risk of node failures and improve the cluster’s stability.
Dedicated master nodes are usually set up in a separate availability zone (AZ) to ensure high availability and reduce the risk of data loss in case of failure. It is recommended to use an odd number of dedicated master nodes, such as 3, 5, or 7, to avoid split-brain scenarios, where the cluster may become unstable if there is an equal number of master nodes.
Enable Auto-Tune
Auto-Tune is a feature of Amazon OpenSearch Service that uses machine learning algorithms to automatically optimize the performance of the search and analytics cluster based on workload patterns. It analyzes the cluster’s usage patterns and suggests changes to the configuration settings to improve performance and reduce management overhead.
Enabling Auto-Tune is a simple process that can be done through the Amazon OpenSearch Service console or the AWS CLI. Once enabled, Auto-Tune continuously monitors the cluster’s performance and adjusts the configuration settings as needed. It can also detect anomalies and notify the cluster administrators of any issues that require attention.
Features and use cases
Auto-Tune automatically adjusts various cluster settings, such as the JVM heap size, the number of shards per node, and the replica settings, based on the observed workload patterns. It analyzes cluster performance metrics such as CPU usage, heap usage, and query response times, and suggests configuration changes to improve performance.
Auto-Tune is particularly useful for workloads that have variable or unpredictable traffic patterns, or for clusters that are deployed for the first time and need to be tuned. It can help reduce the time and effort required to optimize the cluster’s performance, and can ensure that the cluster is always running at peak efficiency.
Sizing Amazon OpenSearch Service Domains
Amazon OpenSearch Service provides a range of instance types and storage options to support different use cases and workloads. It is recommended to use the Amazon OpenSearch Service Console or the AWS CLI to estimate the size of the domain based on the following factors:
- Data volume: The amount of data that needs to be stored in the cluster will impact the size of the domain. It is important to choose the right instance types and storage options to support the data volume.
- Query complexity: Complex queries that require many fields, facets, and filters can impact the performance of the cluster. It is important to consider the complexity of the queries when sizing the domain.
- Query throughput: The number of queries per second that the cluster needs to support will impact the size of the domain. It is important to choose the right instance types and storage options to support the expected query throughput.
- Indexing rate: The rate at which data needs to be indexed in the cluster will impact the size of the domain. It is important to choose the right instance types and storage options to support the expected indexing rate. It is also important to consider the future growth of the data and the need to scale the cluster up or down based on changing requirements.
In general, it is recommended to start with a smaller domain size and scale up as needed based on workload patterns. Amazon OpenSearch Service provides auto-scaling capabilities that can automatically add or remove nodes based on demand, which can help optimize cost and performance.
Monitoring and Alerting
Amazon OpenSearch Service provides various monitoring and alerting tools to help you monitor the health, performance, and availability of your cluster, including:
- Amazon CloudWatch: This monitoring service provides metrics and logs for various AWS resources, including OpenSearch Service clusters. CloudWatch can monitor the cluster and notify administrators when certain thresholds are exceeded.
- OpenSearch Service Console: The console provides monitoring and troubleshooting tools that can help analyze cluster metrics, view the cluster’s state, and monitor the health of nodes.
- AWS CloudTrail: This logging service can record API calls and events that occur in OpenSearch Service clusters. This information can help administrators understand usage patterns, detect security threats, and troubleshoot issues.
Learn more in our detailed guide to :