In response to the cloud service provider, Elastic recently made changes to the official Python client for its Elasticsearch database (Elasticsearch-py) to make it incompatible with the forks, and then brutally shut down comments on the topic on GitHub. This action caused a heated discussion among developers.
Sword pointing to cloud vendors
Elasticsearch is a database manager and analytics engine that is widely used in the industry. The official client is available in Java, .NET (C#), PHP, Python, Apache Groovy, Ruby and many other languages. According to DB-Engines rankings, Elasticsearch is the most popular enterprise search engine, followed by Apache Solr.
Elasticsearch-py aims to provide a consensus for all Elasticsearch-related code in Python, and has been downloaded more than 202,000 times by the client.
Elasticsearch-py has maintained a fundamental position of neutrality and high scalability, and Elasticsearch DSL, the high-level library responsible for running Elasticsearch queries, uses Elasticsearch-py as a library.
This code change is also a reflection of the intensification of the conflict between Elastic and AWS.
As an open source product, Elasticsearch changed its open source license in January of this year from the previous Apache 2.0 license to a dual license model (i.e. SSPL 1.0 and Elastic license), allowing users to choose the license that suits them. The biggest reason for Elastic’s decision was to address unfair usage on public cloud platforms, specifically AWS.
“As many companies continue to transition to SaaS, some cloud service providers are using open source products and offering them as a service to the public without providing anything back to the community. This practice diverts money that could be reinvested in the product, to the detriment of users and the community.” In a statement, Elastic wrote that “the community is coming to realize that open source companies can only maintain high levels of investment and innovation if they better protect their software.”
In response to the change, AWS forked Elasticsearch ahead of the license change, building up Open Distro for Elasticsearch, which has since evolved into OpenSearch and was released as version 1.0 last month.
According to AWS, OpenSearch is a community-driven, open source search and analytics suite derived from Elasticsearch 7.10.2 and Kibana 7.10.2 licensed under Apache 2.0. It includes a search engine daemon (OpenSearch), a visualization and user interface (OpenSearch Dashboards), and Open Distro for elastic search, including security, alerts, anomaly detection, and other features.
According to Adrian Cockcroft, vice president of Amazon Web Services, the release notes and documentation fail to clarify what is and is not open source, leaving enterprise developers in a situation where they can inadvertently use code that could cause financial or legal problems in the future. AWS’ involvement ensures that its customers can continue to run Elasticsearch without worrying that their billing cycle might be interrupted. Some developers, however, say the OpenSearch community activity has yet to increase.
Now, developers have noticed that the source code for Elasticsearch-py has been quietly changed to separately check whether the database is an Elastic or fork product. The update notes that “If the response does not contain the X-Elastic-Product HTTP header, or if the value of the X-Elastic-Product HTTP header is not Elasticsearch, an error will be raised.”
Forcing developers to take sides?
“When the gods fight, the mortals suffer”. This inter-company confrontation has deeply hurt the open source developers who have contributed to Elasticsearch. In the view of Lars Holm Nielsen, product manager at Invenio, a data search management open source project, Elastic is forcing ordinary developers to take sides.
“We have developed an open source product that can easily be used with Elasticsearch or OpenSearch, and users then choose whether to use Elasticsearch or OpenSearch depending on their needs …… Elastic’s actions have really undermined our confidence in the company and its products. Don’t blame Amazon. Elastic has already changed its server licenses, so there’s no need to shut out other forks.” Nielsen said.
Philip Krauss, senior engineering manager at Elastic, responded, “Amazon OpenSearch is a different product. While it has some roots in Elasticsearch, the many differences between the two are bound to cause a lot of problems and even confusion.”
The comment feature on GitHub for the topic has now been turned off, and follow-up messages have been removed.
NET Connector for Elasticsearch was not immune to the changes, and errors such as “The client found that the server is not a supported Elasticsearch distribution” began to appear.
In response to user complaints, Elastic senior software engineer Steve Gordon said, “We recommend that you upgrade to the latest default distribution of Elasticsearch, which is available for free under the Elastic License v2… …We view this change as an enhancement because it only affects unsupported client-server combinations. In supported configurations, the change will not cause any business impact. The purpose of this tweak is to declare incompatibility by failing fast and avoiding the false belief by consumers that they can run loads for long periods of time in configurations that are untested and may not achieve the desired results.”
There’s also one more change: the Java client for Elasticsearch has also switched to the Elastic License, an issue that has sparked anxiety among users in the OpenSearch community.
“How is OpenSearch going to handle the multiple connectors and bindings for the various programming languages currently available, many of which are reportedly already integrated with anti-competitive mechanisms?” One developer questioned.
Licensing constraints are not yet the same thing as product inspection, and Elastic says, “Our client library still follows the Apache 2.0 license, but it does not include the Java High-Level Rest Client (Java HLRC). Over time, we will phase out this dependency and move Java HLRC to the Apache 2.0 license.”
These clients that follow the Apache 2.0 license (including Python and .NET clients) will not work with OpenSearch if connectivity is blocked at the code level. Of course, forking and modifying each client is not that difficult, and there should still be a workaround.
The battle of interests
The rise of the cloud gives basic software companies and open source companies a good distribution channel to better build a competitive barrier, but also to quickly develop open source users to the cloud. Unified deployment on the cloud eliminates the high cost of companies having to install, deploy and even customize for each user. But this has created an impact on the traditional business model of open source software companies.
In recent years, the conflict between cloud vendors and open source vendors has become increasingly pronounced. As early as late ‘18, Neo4j, a graph database, announced that starting with Neo4j version 3.5, the enterprise version would only be available under a commercial license and would no longer be available as source code. In recent years, Confluent, MongoDB, Kafka, Redis, and others have also modified their open source protocols to restrict cloud vendors from profiting from them without contributing.
Developers have different perspectives on the dispute between the two camps.
“I personally can’t stand Elastic either because they charge for enterprise level support and then provide open source level support. The response you get when you have a problem is usually ‘Why try to do that?’ , or ‘Please refer to this question that hasn’t been new since 2016.” Some code contributors shared their own feelings about using Elastic.
As competition intensifies, the commercial companies behind open source software may have to consider how to evolve their services and business models.
However, Amanda Brock, CEO and chief policy officer at Open UK, believes that open source is diverse, but it is not a business model. Companies like Elastic that are unhappy with the way their code is used by cloud providers don’t fully understand the implications of open source licensing. “In my experience, cloud provider vendors are using it in a way that is acceptable within the scope of an open source license.”
Of course, there are developers who say they understand the approach of Elastic’s open source vendors.
If Elastic is successful with ElasticSearch, then it is entirely expected that other companies will join the trend and try to profit from it. As the company creating the product may not be the company that can profit the most from it, companies should plan to make enough profit. But where things get complicated is that no company wants to make several orders of magnitude less from the product they create than other companies that rely on that product.
Open source software companies didn’t foresee the emergence of cloud service delivery vendors that minimized their value proposition. Amazon could have invested significant resources, perhaps even more than Elastic itself. Some might argue that Amazon abused their monopoly in the cloud services space by offering a bundled ElasticSearch experience that Elastic could never compete with. Even if they developed an alternative on-premise product, it would look exactly the same as when Microsoft bundled Internet Explorer with Windows.
Even today, if a company is willing to develop an open source or free software product, it is likely to become a target for exploitation by large enterprises once it is successful enough. Companies always have the option to relicense code written in-house, as well as any code submitted by signatories to the appropriate Contributor License Agreement (CLA).
The issue of benefit sharing is really the biggest point of contention between open source vendors and cloud vendors, and this issue may still have to be resolved from a business perspective to see who can get the upper hand in the game between the two.