Resolving the Elasticsearch index read-only 403 issue

The stack information of the Java exception log is as follows:

2022-03-30 18:17:02.750 ERROR 22991 --- [task-6] .a.i.SimpleAsyncUncaughtExceptionHandler : Unexpected exception occurred invoking async method: public void com.proinnova.system.log.service.impl.LogServiceImpl.addLog(com.proinnova.system.log.po.SysLogInfo,java.lang.Exception)
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=cluster_block_exception, reason=index [os_exception_log] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
        at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2061) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2031) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1778) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1735) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1697) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:929) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.index(ElasticsearchRestTemplate.java:703) ~[spring-data-elasticsearch-3.2.12.RELEASE.jar!/:3.2.12.RELEASE]
        at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.save(AbstractElasticsearchRepository.java:182) ~[spring-data-elasticsearch-3.2.12.RELEASE.jar!/:3.2.12.RELEASE]
        at sun.reflect.GeneratedMethodAccessor204.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_301]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_301]
        at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:371) ~[spring-data-commons-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:204) ~[spring-data-commons-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:658) ~[spring-data-commons-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.12.RELEASE.jar!/:5.2.12.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:622) ~[spring-data-commons-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:606) ~[spring-data-commons-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.12.RELEASE.jar!/:5.2.12.RELEASE]

Main error content:

org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=cluster_block_exception, reason=index [os_exception_log] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

The query is that the ES index is read-only, and the next step is to analyze why the es index is read-only

ES index read-only reason

Insufficient memory When JVMMemoryPressure exceeds 92% and lasts for 30 minutes, ES triggers a protection mechanism and blocks write operations to prevent the cluster from reaching a red state. IndexCreateBlockException , when the recovery is less than 88% in five minutes, write protection will be disabled.
Insufficient disk space The default disk level alert for es is 85%, once the disk usage exceeds 85%, es will not allocate a slice for that node, es also has a disk level alert of 90%, after that, it will try to relocate the slice to another node.

Troubleshooting

Next we check the ES index status to see if it is read-only.

#首先获取索引列表
[root@abcdocker ~]# curl -XGET 'localhost:9200/_cat/indices?v=true&s=index' 
health status index              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   job_content_sunsea 5-wLLa6YQ_-IEqSNrYfY0w   1   1        611            0    232.7kb        232.7kb
yellow open   job_current_sunsea aXz5NNEFRpuPFGMfKdkzbw   1   1         10            0    107.7kb        107.7kb
yellow open   os_exception_log   zH8f77gLTemzBnUBSFPt7w   5   1        191            0      504kb          504kb
yellow open   space_log          ePVDkDhYRaO1KvPibBKbRQ   5   1      32667            0      2.6mb          2.6mb

Look inside the index and check if the read_only_allow_delete variable is true.

[root@abcdocker ~]# curl -s -XGET 'localhost:9200/space_log/_settings' |grep read_only_allow_delete
{"space_log":{"settings":{"index":{"refresh_interval":"1s","number_of_shards":"5","blocks":{"read_only_allow_delete":"true"},"provided_name":"space_log","creation_date":"1639476549574","store":{"type":"fs"},"number_of_replicas":"1","uuid":"ePVDkDhYRaO1KvPibBKbRQ","version":{"created":"7020099"}}}}}
[root@abcdocker ~]# curl -s -XGET 'localhost:9200/os_exception_log/_settings' |grep read_only_allow_delete
{"os_exception_log":{"settings":{"index":{"refresh_interval":"1s","number_of_shards":"5","blocks":{"read_only_allow_delete":"true"},"provided_name":"os_exception_log","creation_date":"1639476469103","store":{"type":"fs"},"number_of_replicas":"1","uuid":"zH8f77gLTemzBnUBSFPt7w","version":{"created":"7020099"}}}}}
[root@abcdocker ~]# curl -s -XGET 'localhost:9200/job_current_sunsea/_settings' |grep read_only_allow_delete
{"job_current_sunsea":{"settings":{"index":{"number_of_shards":"1","blocks":{"read_only_allow_delete":"true"},"provided_name":"job_current_sunsea","creation_date":"1640141880762","number_of_replicas":"1","uuid":"aXz5NNEFRpuPFGMfKdkzbw","version":{"created":"7020099"}}}}}
[root@abcdocker ~]# curl -s -XGET 'localhost:9200/job_current_sunsea/_settings' |grep read_only_allow_delete
{"job_current_sunsea":{"settings":{"index":{"number_of_shards":"1","blocks":{"read_only_allow_delete":"true"},"provided_name":"job_current_sunsea","creation_date":"1640141880762","number_of_replicas":"1","uuid":"aXz5NNEFRpuPFGMfKdkzbw","version":{"created":"7020099"}}}}}

Found that all indexes on es become read_only_allow_delete=true

Solution

The first one: use curl command to change all read-only status to false

1
2
3

curl -XPUT -H "Content-Type: application/json" \
   http://localhost:9200/_all/_settings \
     -d '{"index.blocks.read_only_allow_delete": false}'

Second: Execute the following command inside the Operations tool of the kibana console

1
2

PUT /_all/_settings
{"index.blocks.read_only_allow_delete": false}

Check

kibana console

Table of Contents

ES index read-only reason

Troubleshooting

Solution