Serhii KorolFirst words
I want to discuss data objects such as JSON, BSON, and YAML in this article. We all use JSON to store and pass data through different APIs. I can assure you that the JSON doesn't have the most optimal variant, but it is possible to improve, and I tell you how.
Environment
First of above, let's create a new project.
dotnet new console -n DataObjectsSample
I generated a random JSON object and saved it to the project.
[
{
"_id": "64bb8c0a2217742f16b88662",
"index": 0,
"guid": "d9d8fcc2-f46b-4089-a7ea-45472ee82e2b",
"isActive": true,
"balance": "$2,590.27",
"picture": "http://placehold.it/32x32",
"age": 40,
"eyeColor": "blue",
"name": "Cameron Burt",
"gender": "male",
"company": "SONGLINES",
"email": "cameronburt@songlines.com",
"phone": "+1 (839) 567-3083",
"address": "993 Virginia Place, Genoa, Colorado, 4207",
"about": "Adipisicing laboris amet reprehenderit fugiat magna voluptate amet Lorem voluptate proident enim est non anim. Quis pariatur labore velit proident mollit eiusmod ipsum voluptate amet aute ad consequat deserunt. Ipsum aute laborum dolor laborum incididunt deserunt eiusmod mollit exercitation proident et. Ullamco adipisicing ad veniam laborum ea adipisicing exercitation elit laborum in nisi irure aliqua reprehenderit.\r\n",
"registered": "2020-09-08T11:06:52 -03:00",
"latitude": -52.585211,
"longitude": 87.57059,
"tags": [
"magna",
"dolor",
"exercitation",
"est",
"enim",
"excepteur",
"duis"
],
"friends": [
{
"id": 0,
"name": "Annmarie Beasley"
},
{
"id": 1,
"name": "Davis Bentley"
},
{
"id": 2,
"name": "Nora Watkins"
}
],
"greeting": "Hello, Cameron Burt! You have 8 unread messages.",
"favoriteFruit": "apple"
},
{
"_id": "64bb8c0aa9cb4d918bc20239",
"index": 1,
"guid": "e4f89fef-c035-4bd6-a177-9bf791c5d0c4",
"isActive": true,
"balance": "$1,825.66",
"picture": "http://placehold.it/32x32",
"age": 31,
"eyeColor": "brown",
"name": "Vinson Jacobson",
"gender": "male",
"company": "SLUMBERIA",
"email": "vinsonjacobson@slumberia.com",
"phone": "+1 (863) 446-2500",
"address": "215 Vanderbilt Street, Rockhill, Massachusetts, 6585",
"about": "Id duis id do esse incididunt culpa do sit ea duis proident dolore sit. Eu sit aliquip exercitation in. Nostrud aliquip dolore culpa est minim minim sint. Dolor laborum irure pariatur cillum labore deserunt do ad veniam consequat duis culpa ut quis. Sit pariatur cupidatat dolore excepteur dolore aliquip ea reprehenderit incididunt minim deserunt. Pariatur elit est laborum in ullamco qui sunt.\r\n",
"registered": "2017-01-31T07:10:59 -02:00",
"latitude": 77.910014,
"longitude": 115.80198,
"tags": [
"minim",
"consectetur",
"quis",
"eiusmod",
"velit",
"esse",
"id"
],
"friends": [
{
"id": 0,
"name": "Katharine Hooper"
},
{
"id": 1,
"name": "Maggie Branch"
},
{
"id": 2,
"name": "Etta Bradford"
}
],
"greeting": "Hello, Vinson Jacobson! You have 1 unread messages.",
"favoriteFruit": "apple"
},
{
"_id": "64bb8c0a504a35a2e9beb72c",
"index": 2,
"guid": "0284317b-078e-446e-8e82-9cc880e60570",
"isActive": true,
"balance": "$3,319.64",
"picture": "http://placehold.it/32x32",
"age": 33,
"eyeColor": "green",
"name": "Acosta Hatfield",
"gender": "male",
"company": "EBIDCO",
"email": "acostahatfield@ebidco.com",
"phone": "+1 (844) 521-2227",
"address": "541 Glenmore Avenue, Hailesboro, Mississippi, 7771",
"about": "Reprehenderit anim irure excepteur elit nulla nisi est excepteur labore sunt tempor. Sit nisi sit officia sit ea id mollit duis deserunt magna amet. Voluptate ad voluptate laboris veniam occaecat est est occaecat sint pariatur. Esse laborum aute adipisicing aliqua id nostrud nisi nostrud ullamco. Do voluptate commodo labore quis veniam deserunt quis excepteur quis esse. Duis sunt fugiat ex adipisicing ipsum excepteur cupidatat amet ipsum commodo amet. Nisi do ex excepteur Lorem irure velit voluptate commodo aliquip adipisicing aliqua.\r\n",
"registered": "2015-07-15T08:39:58 -03:00",
"latitude": 56.346667,
"longitude": -109.26569,
"tags": [
"est",
"laborum",
"occaecat",
"culpa",
"labore",
"id",
"aliquip"
],
"friends": [
{
"id": 0,
"name": "Nelda Sullivan"
},
{
"id": 1,
"name": "Ina Ferrell"
},
{
"id": 2,
"name": "Ayers Villarreal"
}
],
"greeting": "Hello, Acosta Hatfield! You have 10 unread messages.",
"favoriteFruit": "apple"
},
{
"_id": "64bb8c0a827c2c0a43f59bbe",
"index": 3,
"guid": "819071de-08d0-4001-a3cd-0e0f77395874",
"isActive": false,
"balance": "$3,317.60",
"picture": "http://placehold.it/32x32",
"age": 39,
"eyeColor": "blue",
"name": "Nettie Duke",
"gender": "female",
"company": "ZANITY",
"email": "nettieduke@zanity.com",
"phone": "+1 (860) 435-3370",
"address": "283 Homecrest Court, Craig, New Jersey, 1613",
"about": "Nulla ex mollit cillum ut qui officia. Enim sunt fugiat dolore incididunt. Dolore nostrud consequat est occaecat.\r\n",
"registered": "2020-05-19T01:22:42 -03:00",
"latitude": 42.407572,
"longitude": -131.36768,
"tags": [
"tempor",
"in",
"enim",
"consectetur",
"incididunt",
"amet",
"non"
],
"friends": [
{
"id": 0,
"name": "Petersen Harrington"
},
{
"id": 1,
"name": "Vaughn Sargent"
},
{
"id": 2,
"name": "Estela Reeves"
}
],
"greeting": "Hello, Nettie Duke! You have 3 unread messages.",
"favoriteFruit": "strawberry"
},
{
"_id": "64bb8c0ae84328d94ca8d531",
"index": 4,
"guid": "f33531c5-b91e-4a7a-84db-aa4b6488af5a",
"isActive": true,
"balance": "$3,349.83",
"picture": "http://placehold.it/32x32",
"age": 32,
"eyeColor": "green",
"name": "Levy Williamson",
"gender": "male",
"company": "AQUASURE",
"email": "levywilliamson@aquasure.com",
"phone": "+1 (816) 471-2716",
"address": "645 Cooke Court, Fairfield, Georgia, 6860",
"about": "Consequat pariatur mollit consectetur Lorem nisi et fugiat laboris minim dolor ex cillum nisi. Ut laboris laborum do eiusmod et eu Lorem exercitation eiusmod et dolor. Id ut incididunt irure reprehenderit aute nulla officia aute ad. Velit deserunt ullamco eiusmod dolore elit deserunt ullamco eu qui proident nulla voluptate sit occaecat. Quis aliquip nisi laborum sint mollit do. Duis fugiat et minim exercitation fugiat aliqua ea quis proident duis proident aute. Dolore do culpa ex tempor dolor consequat non nulla ut veniam adipisicing.\r\n",
"registered": "2022-10-11T08:34:11 -03:00",
"latitude": 88.921132,
"longitude": 107.27617,
"tags": [
"nostrud",
"qui",
"consequat",
"ipsum",
"duis",
"nostrud",
"exercitation"
],
"friends": [
{
"id": 0,
"name": "Genevieve Boyle"
},
{
"id": 1,
"name": "Reyna Knapp"
},
{
"id": 2,
"name": "Coffey Wiley"
}
],
"greeting": "Hello, Levy Williamson! You have 10 unread messages.",
"favoriteFruit": "banana"
}
]
And also, for more convenience, you can add this code to DataObjectsSample.csproj:
<ItemGroup>
<None Update="generated.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>
And now add the start template to Program.cs:
internal static class Program
{
private static void Main()
{
string currentDirectory = Directory.GetCurrentDirectory();
string filePath = Path.Combine(currentDirectory, "generated.json");
}
}
JSON
For JSON no need any packages. You need to read the text file and show it in a console. Add this code:
string json = File.ReadAllText(filePath);
Console.WriteLine($"Original json size: {json.Length} Bytes");
You should see in the console like this:
Original json size: 7251 Bytes
Is this much or few? We'll compare different data objects with the same data.
BSON
Probably many people heard about the BSON format. However, I split it into two parts since they work with different performance. I'll convert the string to BSON with NewtonSoft. Add this row to your project:
<PackageReference Include="Newtonsoft.Json.Bson" Version="1.0.3-beta1" />
And let's implement BSON converting:
string bson = GetBson(json);
Console.WriteLine($"Original bson size: {bson.Length} Bytes");
private static string GetBson(string json)
{
var jsonObj = JsonConvert.DeserializeObject(json);
var ms = new MemoryStream();
using (var writer = new BsonWriter(ms))
{
JsonSerializer serializer = new JsonSerializer();
serializer.Serialize(writer, jsonObj);
}
return Convert.ToBase64String(ms.ToArray());
}
If you run it, you will see the size is 1000 bytes larger.
Original bson size: 8288 Bytes
It's the first reason don't use this approach for BSON. And the second reasonб it is obsolete. However, there is a better way to convert to BSON. You need to add this package:
<PackageReference Include="MongoDB.Bson" Version="2.20.0" />
And add similar code to your project:
string mongo = BsonArray(json);
Console.WriteLine($"Original mongo size: {mongo.Length} Bytes");
private static string BsonArray(string json)
{
using var jsonReader = new MongoDB.Bson.IO.JsonReader(json);
var serializer = new MongoDB.Bson.Serialization.Serializers.BsonArraySerializer();
var bsonArray = serializer.Deserialize(BsonDeserializationContext.CreateRoot(jsonReader));
return bsonArray.ToString();
}
Let's check and result much better:
Original mongo size: 6312 Bytes
YAML
And the last format that I want to show is YAML. For this you also need the package:
<PackageReference Include="YamlDotNet" Version="13.1.1" />
And now make it:
string yaml = ToYaml(json);
Console.WriteLine($"Original yaml size: {yaml.Length} Bytes");
private static string ToYaml(string json)
{
var document = ConvertJTokenToObject(JsonConvert.DeserializeObject<JToken>(json)!);
var serializer = new YamlDotNet.Serialization.Serializer();
using var writer = new StringWriter();
serializer.Serialize(writer, document);
var yaml = writer.ToString();
return yaml;
static object? ConvertJTokenToObject(JToken token)
{
return token switch
{
JValue value => value.Value,
JArray => token.AsEnumerable().Select(ConvertJTokenToObject).ToList(),
JObject => token.AsEnumerable()
.Cast<JProperty>()
.ToDictionary(x => x.Name, x => ConvertJTokenToObject(x.Value)),
_ => throw new InvalidOperationException("Unexpected token: " + token)
};
}
}
You likely detected this ConvertJTokenToObject method and it's need for filling YAML with values. Otherwise, you'll get YAML only with properties and without values.
Let's check it and it'll better result:
Original yaml size: 5799 Bytes
Improvements
Seeing these results, you can ask that you no longer need to use JSON? Is it so heavyweight? Yes, if you send massive objects, you likely need to consider other objects. However, I'll show you how to improve it. We can compress objects as we tend to compress email attachments. If for ordinary files we use ZIP, then for data transporting, we'll be using GZIP.
Add this code:
byte[] jsonCompressed = CompressStringData(json);
byte[] bsonCompressed = CompressStringData(bson);
byte[] mongoCompressed = CompressStringData(mongo);
byte[] yamlCompressed = CompressStringData(yaml);
Console.WriteLine($"Compressed json size: {jsonCompressed.Length} Bytes");
Console.WriteLine($"Compressed bson size: {bsonCompressed.Length} Bytes");
Console.WriteLine($"Compressed mongo size: {mongoCompressed.Length} Bytes");
Console.WriteLine($"Compressed yaml size: {yamlCompressed.Length} Bytes");
private static byte[] CompressStringData(string jsonData)
{
byte[] byteArray = Encoding.UTF8.GetBytes(jsonData);
using var memoryStream = new MemoryStream();
using (var gzipStream = new GZipStream(memoryStream, CompressionLevel.Optimal))
{
gzipStream.Write(byteArray, 0, byteArray.Length);
}
return memoryStream.ToArray();
}
If you'll check it out then you'll see that JSON isn't almost the worst.
Original json size: 7251 Bytes
Original bson size: 8288 Bytes
Original mongo size: 6312 Bytes
Original yaml size: 5799 Bytes
Compressed json size: 2354 Bytes
Compressed bson size: 3865 Bytes
Compressed mongo size: 2308 Bytes
Compressed yaml size: 2247 Bytes
Conclusion
If you use huge JSONs then optimize them or choose other formats.
Thank you for reading and see you in the next article. Happy coding! The source code you can find by this link.
