Hello ๐๏ธ !
Following the mechanisms described here to send CSV files into Kafka in JSON using kcat through shell pipelines, we packed ๐ฆ all of that in a new home-made tool named producer-file
. Here are the features :
- ๐จ Sending line by line
- ๐ช CSV to JSON conversion
- ๐ฆ Support message headers by file and/or by line
- โก๏ธ Delta-changes mode (only send changes occurred after last send)
โ๏ธ More about the shell way :
โจ๏ธ Pipe xlsx files into/from Kafka... From cli with (k)cat ๐
adriens for opt-nc ใป Oct 10 '22 ใป 2 min read
Motivations
Using jq
+ kcat
with shell pipelines do the job well, but this doesn't fit the company's requirements today about software packaging and logging capabilities ๐คท๐ปโโ๏ธ.
๐ฝ๏ธ Demo
Short video demonstrating how to use this tool to send a CSV file in JSON into Kafka within a Docker environnement :