Cannot create policy after upgrading to v1.12

[adityaj1107]

Issue by tocw
Wednesday Jan 20, 2021 at 10:05 GMT
Originally opened as opendistro-for-elasticsearch/index-management#388

---

Hello,
after upgrading to 1.12 I noticed we are not able to create ISM policies.
I simply took the example from your documentation here and this is the error message I got:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_exception",
        "reason" : "the [enabled] parameter can't be updated for the object mapping [policy.states.actions.allocation.exclude]"
      }
    ],
    "type" : "mapper_exception",
    "reason" : "the [enabled] parameter can't be updated for the object mapping [policy.states.actions.allocation.exclude]"
  },
  "status" : 500
}

We run pure 1.12 release with no additional modifications. Please investigate as this is a critical issue for our production clusters

[adityaj1107]

Comment by dbbaughe
Wednesday Jan 20, 2021 at 16:03 GMT

---

Hi @tocw,

Can you paste the mappings of your .opendistro-ism-config index.

[adityaj1107]

Comment by samling
Wednesday Jan 20, 2021 at 16:27 GMT

---

We ran into this same issue when upgrading to 1.12, we had been using our own fork of the ISM plugin with the allocation action as implemented in I believe opendistro-for-elasticsearch/index-management#106 before it was officially merged. It's unclear to me whether this code changed in the final release, but in any case, we were able to resolve it with the following steps, most easily performed in the kibana dev console:

1. GET .opendistro-ism-config
2. Copy the contents to some editor
3. Remove the entire allocation{} block under mappings.properties.policy.states.properties.actions.properties (easier to just ctrl-f for 'allocation')
4. DELETE .opendistro-ism-config
5. PUT .opendistro-ism-config with the modified json
6. Try creating your ISM policy

Note that you probably need to get rid of the top-level key ({ ".opendistro-ism-config": ...}) before step 5 will work. The allocation block in the ISM config will be recreated with the correct parameters.

[adityaj1107]

Comment by tocw
Thursday Jan 21, 2021 at 09:53 GMT

---

Hi @tocw,

Can you paste the mappings of your .opendistro-ism-config index.

Hello @dbbaughe ,

here are the mappings:

{
   "dynamic" : "strict",
   "_meta" : {
      "schema_version" : 3
   },
   "properties" : {
      "managed_index" : {
         "properties" : {
            "change_policy" : {
               "properties" : {
                  "is_safe" : {
                     "type" : "boolean"
                  },
                  "policy_id" : {
                     "type" : "keyword"
                  },
                  "state" : {
                     "type" : "keyword"
                  }
               }
            },
            "enabled" : {
               "type" : "boolean"
            },
            "enabled_time" : {
               "type" : "date",
               "format" : "strict_date_time||epoch_millis"
            },
            "index" : {
               "type" : "keyword"
            },
            "index_uuid" : {
               "type" : "keyword"
            },
            "last_updated_time" : {
               "type" : "date",
               "format" : "strict_date_time||epoch_millis"
            },
            "name" : {
               "type" : "text",
               "fields" : {
                  "keyword" : {
                     "type" : "keyword",
                     "ignore_above" : 256
                  }
               }
            },
            "policy" : {
               "type" : "object",
               "enabled" : false
            },
            "policy_id" : {
               "type" : "keyword"
            },
            "policy_primary_term" : {
               "type" : "long"
            },
            "policy_seq_no" : {
               "type" : "long"
            },
            "schedule" : {
               "properties" : {
                  "cron" : {
                     "properties" : {
                        "expression" : {
                           "type" : "keyword"
                        },
                        "timezone" : {
                           "type" : "keyword"
                        }
                     }
                  },
                  "interval" : {
                     "properties" : {
                        "period" : {
                           "type" : "integer"
                        },
                        "start_time" : {
                           "type" : "date",
                           "format" : "strict_date_time||epoch_millis"
                        },
                        "unit" : {
                           "type" : "keyword"
                        }
                     }
                  }
               }
            }
         }
      },
      "policy" : {
         "properties" : {
            "default_state" : {
               "type" : "keyword"
            },
            "description" : {
               "type" : "text",
               "fields" : {
                  "keyword" : {
                     "type" : "keyword",
                     "ignore_above" : 256
                  }
               }
            },
            "error_notification" : {
               "properties" : {
                  "destination" : {
                     "properties" : {
                        "chime" : {
                           "properties" : {
                              "url" : {
                                 "type" : "text",
                                 "fields" : {
                                    "keyword" : {
                                       "type" : "keyword",
                                       "ignore_above" : 256
                                    }
                                 }
                              }
                           }
                        },
                        "custom_webhook" : {
                           "properties" : {
                              "header_params" : {
                                 "type" : "object",
                                 "enabled" : false
                              },
                              "host" : {
                                 "type" : "text"
                              },
                              "password" : {
                                 "type" : "text"
                              },
                              "path" : {
                                 "type" : "keyword"
                              },
                              "port" : {
                                 "type" : "integer"
                              },
                              "query_params" : {
                                 "type" : "object",
                                 "enabled" : false
                              },
                              "scheme" : {
                                 "type" : "keyword"
                              },
                              "url" : {
                                 "type" : "text",
                                 "fields" : {
                                    "keyword" : {
                                       "type" : "keyword",
                                       "ignore_above" : 256
                                    }
                                 }
                              },
                              "username" : {
                                 "type" : "text"
                              }
                           }
                        },
                        "last_update_time" : {
                           "type" : "date",
                           "format" : "strict_date_time||epoch_millis"
                        },
                        "name" : {
                           "type" : "text",
                           "fields" : {
                              "keyword" : {
                                 "type" : "keyword",
                                 "ignore_above" : 256
                              }
                           }
                        },
                        "slack" : {
                           "properties" : {
                              "url" : {
                                 "type" : "text",
                                 "fields" : {
                                    "keyword" : {
                                       "type" : "keyword",
                                       "ignore_above" : 256
                                    }
                                 }
                              }
                           }
                        },
                        "type" : {
                           "type" : "keyword"
                        }
                     }
                  },
                  "message_template" : {
                     "type" : "object",
                     "enabled" : false
                  }
               }
            },
            "last_updated_time" : {
               "type" : "date",
               "format" : "strict_date_time||epoch_millis"
            },
            "policy_id" : {
               "type" : "text",
               "fields" : {
                  "keyword" : {
                     "type" : "keyword",
                     "ignore_above" : 256
                  }
               }
            },
            "schema_version" : {
               "type" : "long"
            },
            "states" : {
               "type" : "nested",
               "properties" : {
                  "actions" : {
                     "type" : "nested",
                     "properties" : {
                        "allocation" : {
                           "dynamic" : "true",
                           "properties" : {
                              "exclude" : {
                                 "type" : "object"
                              },
                              "include" : {
                                 "type" : "object"
                              },
                              "require" : {
                                 "properties" : {
                                    "box_type" : {
                                       "type" : "text",
                                       "fields" : {
                                          "keyword" : {
                                             "type" : "keyword",
                                             "ignore_above" : 256
                                          }
                                       }
                                    }
                                 }
                              },
                              "wait_for" : {
                                 "type" : "boolean"
                              }
                           }
                        },
                        "close" : {
                           "type" : "object"
                        },
                        "delete" : {
                           "type" : "object"
                        },
                        "force_merge" : {
                           "properties" : {
                              "max_num_segments" : {
                                 "type" : "integer"
                              }
                           }
                        },
                        "index_priority" : {
                           "properties" : {
                              "priority" : {
                                 "type" : "integer"
                              }
                           }
                        },
                        "notification" : {
                           "properties" : {
                              "destination" : {
                                 "properties" : {
                                    "chime" : {
                                       "properties" : {
                                          "url" : {
                                             "type" : "text",
                                             "fields" : {
                                                "keyword" : {
                                                   "type" : "keyword",
                                                   "ignore_above" : 256
                                                }
                                             }
                                          }
                                       }
                                    },
                                    "custom_webhook" : {
                                       "properties" : {
                                          "header_params" : {
                                             "type" : "object",
                                             "enabled" : false
                                          },
                                          "host" : {
                                             "type" : "text"
                                          },
                                          "password" : {
                                             "type" : "text"
                                          },
                                          "path" : {
                                             "type" : "keyword"
                                          },
                                          "port" : {
                                             "type" : "integer"
                                          },
                                          "query_params" : {
                                             "type" : "object",
                                             "enabled" : false
                                          },
                                          "scheme" : {
                                             "type" : "keyword"
                                          },
                                          "url" : {
                                             "type" : "text",
                                             "fields" : {
                                                "keyword" : {
                                                   "type" : "keyword",
                                                   "ignore_above" : 256
                                                }
                                             }
                                          },
                                          "username" : {
                                             "type" : "text"
                                          }
                                       }
                                    },
                                    "last_update_time" : {
                                       "type" : "date",
                                       "format" : "strict_date_time||epoch_millis"
                                    },
                                    "name" : {
                                       "type" : "text",
                                       "fields" : {
                                          "keyword" : {
                                             "type" : "keyword",
                                             "ignore_above" : 256
                                          }
                                       }
                                    },
                                    "slack" : {
                                       "properties" : {
                                          "url" : {
                                             "type" : "text",
                                             "fields" : {
                                                "keyword" : {
                                                   "type" : "keyword",
                                                   "ignore_above" : 256
                                                }
                                             }
                                          }
                                       }
                                    },
                                    "type" : {
                                       "type" : "keyword"
                                    }
                                 }
                              },
                              "message_template" : {
                                 "type" : "object",
                                 "enabled" : false
                              }
                           }
                        },
                        "open" : {
                           "type" : "object"
                        },
                        "read_only" : {
                           "type" : "object"
                        },
                        "read_write" : {
                           "type" : "object"
                        },
                        "replica_count" : {
                           "properties" : {
                              "number_of_replicas" : {
                                 "type" : "integer"
                              }
                           }
                        },
                        "retry" : {
                           "properties" : {
                              "backoff" : {
                                 "type" : "keyword"
                              },
                              "count" : {
                                 "type" : "integer"
                              },
                              "delay" : {
                                 "type" : "keyword"
                              }
                           }
                        },
                        "rollover" : {
                           "properties" : {
                              "min_doc_count" : {
                                 "type" : "keyword"
                              },
                              "min_index_age" : {
                                 "type" : "keyword"
                              },
                              "min_size" : {
                                 "type" : "keyword"
                              }
                           }
                        },
                        "snapshot" : {
                           "properties" : {
                              "include_global_state" : {
                                 "type" : "boolean"
                              },
                              "repository" : {
                                 "type" : "keyword"
                              },
                              "snapshot" : {
                                 "type" : "keyword"
                              }
                           }
                        },
                        "timeout" : {
                           "type" : "keyword"
                        }
                     }
                  },
                  "name" : {
                     "type" : "keyword"
                  },
                  "transitions" : {
                     "type" : "nested",
                     "properties" : {
                        "conditions" : {
                           "type" : "object",
                           "enabled" : false
                        },
                        "state_name" : {
                           "type" : "keyword"
                        }
                     }
                  }
               }
            }
         }
      }
   }
}

[adityaj1107]

Comment by tocw
Thursday Jan 21, 2021 at 09:56 GMT

---

We ran into this same issue when upgrading to 1.12, we had been using our own fork of the ISM plugin with the allocation action as implemented in I believe #106 before it was officially merged. It's unclear to me whether this code changed in the final release, but in any case, we were able to resolve it with the following steps, most easily performed in the kibana dev console:

1. `GET .opendistro-ism-config`

2. Copy the contents to some editor

3. Remove the entire `allocation{}` block under `mappings.properties.policy.states.properties.actions.properties` (easier to just ctrl-f for 'allocation')

4. `DELETE .opendistro-ism-config`

5. `PUT .opendistro-ism-config` with the modified json

6. Try creating your ISM policy

Note that you probably need to get rid of the top-level key ({ ".opendistro-ism-config": ...}) before step 5 will work. The allocation block in the ISM config will be recreated with the correct parameters.

Hello @samling ,
deleting the index is not an option. I have followed your solution on test environment and all existing policies were deleted.

[adityaj1107]

Comment by dbbaughe
Thursday Jan 21, 2021 at 17:07 GMT

---

Hi @tocw, based on your mappings it does seem like you used a fork with the Allocation PR before it was merged in? You can always add more mappings to an index, but you can't modify existing mappings which unfortunately is what you need to do as you have mappings for allocation that don't exist in the plugin and conflict with the plugin's mappings.

You will have to ultimately do some variation of what @samling posted to get back in line with our production release of the plugin. You'll have to make copies of whats in the config index, delete the existing one, and then add the copies back into the config index ensuring you have the correct mappings. You can just create a dummy policy when you want to recreate the index and the plugin will create it all with the correct mappings. The issue you'll run into though is by deleting the index and the jobs the plugin will delete the metadata in the cluster state. You might need to make a bunch of temporary policies that you can re-apply to existing indices that are at different states of their lifecycles to get around it.

[adityaj1107]

Comment by tocw
Friday Jan 22, 2021 at 11:46 GMT

---

Hi @dbbaughe ,
since we have thousands of indices per environment I would like to avoid recreating policies and attaching new policies for indices which have different states. I found out the following solution but I'm not sure if it's safe for production clusters. So far I have tested it on a small test cluster and it seems to be working fine:

1. Create a copy of GET .opendistro-ism-config
2. Fix te mapping problems locally
3. Create a temporary ism config index PUT ism-config using the data from step 2
4. Reindex the data POST _reindex from .opendistro-ism-config to the temporary index
5. Delete the ism config DELETE .opendistro-ism-config
6. Clone the temporary config PUT /ism-config/_clone/.opendistro-ism-config

What do you think about it?