#Elastic filebeats how to
#Elastic filebeats code

Could we document how to set this up in the migration guide? Probably this can more easily achieved by just running filebeat twice with two different configuration files and registry locations. Thoughts We don't have to do it for this PR but we can create a follow up issue and do it separately.įor the purposes of migration it would also be quite nice to define the new filestream and log inputs in the same configuration file and run them next to each other to compare the output. I think this will be important when there are many log inputs in the same Filebeat instance. This will let people be confident that the filestream configuration they wrote only takes over what they actually want it to. I think it should be possible to set the take_over flag and execute it as a dry run where it shows you what paths it would take over without actually doing the migration. I did have one thought while I was testing this to follow up on. Tested locally and LGTM! Thanks! I really like the way this turned out. Closes Automate the migration of log input states to filestream #31749.This verifies that the filestream input took over the state from the log input, didn't duplicate events on startup and ingested new data from the original log file.Īlso, you can see that the registry log was backed up ( /path/to/data/registry/filebeat, see the configs above): 'message': 'filestream inputs took over 1 file(s) from loginputs ', 'message': 'Found the registry log at /Users/rdner/Projects/es_confs/take-over/data/registry/filebeat/log.json ',

'message': 'Checking if the registry log exists at /Users/rdner/Projects/es_confs/take-over/data/registry/filebeat/log.json. 'message': 'Attempting to find the checkpoint.

'message': 'found 1 filestream inputs in `take over` mode ', 'message': 'found 1 patterns for filestream `my-filestream-id` ', 'message': 'recursive glob enabled for filestream `my-filestream-id` ', Stop Filebeat and check the output-filestream.json file created near the Filebeat binary, it should contain log objects with the take over process:.Create a file loginput.log with a few lines like this:.I have added tests that prove my fix is effective or that my feature works.I have made corresponding change to the default configuration files.I have made corresponding changes to the documentation.I have commented my code, particularly in hard-to-understand areas.

#Elastic filebeats code

My code follows the style guidelines of this project.To improve UX of the loginput->filestream migration and allow all our integrations to smoothly migrate to filestream as well. This addresses only the user facing migration from log input to filestream input. This does not address migration of any integration or other input type (like container #34393) that's using loginput under the hood. This solves the previously occurring data duplication (file re-ingestion) problem. All offsets for input files will be preserved and the filestream will continue ingesting the files at the same point where the loginput stopped. The purpose of this mode is to make migration from loginput to filestream as simple and smooth as possible by adding take_over: true to the new filestream configuration. This means the existing loginput state entry gets converted into a filestream entry (the loginput entry gets deleted). If a filestream input has the configuration parameter take_over set to true, every loginput state record (in the registry) with the source that matches at least one of the filestream's paths/globs will be taken over by this filestream input.