Fable 3 made something I was not aware for some time, it moved to emitting ESM Modules and leaving babel and other stuff behind for users to set up.
It was around June that I was really mad at compilation times with Fable projects, After being in the JS/Node ecosystems for years I wondered what could be done to improve that situation.
At the time I had been exploring alternatives to Webpack like fuse-box, parcel, and esbuild. Around the same time I was made aware aware that browsers had already implemented ESM modules, so technically as long as you produced HTML, CSS, and JS you didn't need any kind of pre-processing at all.
This shouldn't be that hard, I just needed a server that well... served the HTML/CSS/JS files right?
I went to my desktop, created an F# script added a couple of libraries like Suave and CliWrap so I could call the dotnet fable command from my F# code and make it compile my Fable files.
Taking out some code I came up with this PoC:
// I omited more code above for brevityletstdinAsyncSeq()=letreadFromStdin()=Console.In.ReadLineAsync()|>Async.AwaitTaskasyncSeq{// I wanted to think this is a "clever"// way to keep it runningwhiletruedolet!value=readFromStdin()value}|>AsyncSeq.distinctUntilChanged|>AsyncSeq.iterAsynconStdinAsyncletapp=choose[path"/">=>GET// send the index file>=>Files.browseFileHome"index.html"// serve static filesGET>=>Files.browseHomeRequestErrors.NOT_FOUND"Not Found"// SPA like fallback>=>redirect"/"]letconfig(publicPath:stringoption)=letpath=Path.GetFullPath(matchpublicPathwith|Some"built"->"./dist"|_->"./public")printfn$"Serving content from {path}"// configure the suave server instance{defaultConfigwithbindings=[HttpBinding.createSimpleHTTP"0.0.0.0"3000]homeFolder=SomepathcompressedFilesFolder=Some(Path.GetFullPath"./.compressed")}// let's make it run!stdinAsyncSeq()|>Async.Start// dotnet fsi suave.fsx built to show how bundled files workstartWebServer(config(fsi.CommandLineArgs|>Array.tryLast))app
Now, I could have my suave server and my Fable compiler running on the background. I could see my files being served in my browser I could make changes, press F5 and see them working.
Cool it worked... Yay!... sure, with my attention span for some things I simply didn't think too much about it, or so I thought.
What came up next was experimenting with snowpack and fuse-box to see which setup could work best with Fable 3 and Although, Both projects work extremely well with Fable, the snowpack project felt more compelling to me thanks to the promoted Unbundled development concept. I decided to go for it and tried the Fable Real World implementation and switched webpack for snowpack and the results were kind of what I was expecting, faster builds, a simpler setup and a much faster developer loop feedback with the browser.
Unconsciously on the back of my head was still that voice about writing something like snowpack in F#... In my mind, the people who build those kinds of tools are like people in the movies; You know they exist but, you don't think you are capable of doing something like it. Specially when most of my experience at that point was building UI's in things like Angular.
I went ahead and started studying the snowpack source code and I found out that they were using esbuild a JS/TS compiler written in Go, no wonder why it was faster than anything done in JavaScript at the time.
Also, on the background vitejs was also starting to get in shape, I was looking at Evan's tweets from afar and getting inspired from that as well so I realized I needed to go back and see if I could go even further.
What if I used esbuild as well?
What if I could use esbuild to produce my prod bundle after I built my fable code?
Turns out... I wasn't that crazy, after all both vite and snowpack were doing it as well!
Around September vite got traction with the vue user base and other users as well. I also studied a bit the vite source code, and even used it for some Fable material for posts. I was trying to make some awareness of Fable.Lit support for Web Components and I wanted to experiment in reality how good vite was, and boi it's awesome If you're starting new projects that depend on node tooling in my opinion, it's your best bet.
Anyways, I tend to be looking at what's new on the web space, and by this time these... Import Maps thing came to my attention, it is a really nice browser feature that can be used to control the browser's behavior to import JavaScript files.
Import maps can tell the browser to use "bare specifiers" (i.e. import dependency from "my-dependency" rather than "./my-dependency.js)
Almost like "pull this import from this URL".
Hopefully you are starting to put the pieces together as I was doing.
Maybe... It might be possible to actually enjoy the NPM ecosystem without having to rely on the local tooling... Just maybe...
From then on, It was just about making small F# scripts to experiment PoC's and implementing small features.
At this point is when I said to myself.
It's time to build a Webpack alternative...
For this... FSharp.DevServer... I needed to have an idea of what I wanted to implement to make it usable at least, I settled on the following set of features.
Serve HTML/CSS/JS
Support for Fable Projects
Reload on change
Install dependencies
Production Bundles
Transpilation on the fly
Dev Proxy
Plugins
HMR
Those are the least features necessary IMO to consider for a project like this.
I will take you on a quick tour at how those got implemented at some point in the project.
Keep in mind I'm not an F# expert! I'm just a guy with a lot of anxiety and free time so I use my code to distract myself while having a ton of fun with F# code.
While for a proof of concept Suave did great, I switched it in favor of Saturn given my familiarity with it and some ASP.NET code.
Serve HTML/CSS/JS
From most of the features, this must have been perhaps the most simple after all if you're using a server, it should be really simple to do right?
Well... Yes and No... it turns out that if you serve static files they get out of the middleware chain very quick due to the order of the static middleware is in. While it was good for serving it if I wanted to reload on change or compile these files I was not going to be able to do it.
// the current devServer function has way more stuff// due the extra features that have been implementedletprivatedevServer(config:FdsConfig)=letwithAppConfig(appConfig:IApplicationBuilder)=// let's serve everything statically// but let's ignore some extensionsletignoreStatic=[".js"".css"".module.css"".ts"".tsx"".jsx"".json"]// mountedDirs is a property in perla.jsonc// that enables you to watch a partcular set of // directories for source codeformapinmountedDirsdoletstaticFileOptions=letprovider=FileExtensionContentTypeProvider()forextinignoreStaticdoprovider.Mappings.Remove(ext)|>ignoreletoptions=StaticFileOptions()options.ContentTypeProvider<-provider// in the next lines we enable local mapings// to URL's e.g.// ./src on disk -> /src on the URLoptions.RequestPath<-PathString(map.Value)options.FileProvider<-newPhysicalFileProvider(Path.GetFullPath(map.Key))optionsappConfig.UseStaticFilesstaticFileOptions|>ignoreletappConfig=// at the same time we enable transpilation// middleware when we're ignoring some extensionsappConfig.UseWhen(Middleware.transformPredicateignoreStatic,Middleware.configureTransformMiddlewareconfig)// set the configured optionsapplication{app_configwithAppConfigwebhost_configwithWebhostConfiguse_endpoint_routerurls}// build itapp.UseEnvironment(Environments.Development).Build()
This part is just about serving files, nothing more, nothing less, that's the core of a dev server.
Support for Fable Projects
Fable is actually not hard to support, fable is distributed as a dotnet tool, we can invoke the command with CliWrap which has proven us in the PoC stage, how simple is to call a process from .NET.
// This is the actual Fable implementationmoduleFable=letmutableprivateactiveFable:intoption=None// this is to start/stop the fable command// if requested by the userletprivatekillActiveProcesspid=tryletactiveProcess=System.Diagnostics.Process.GetProcessByIdpidactiveProcess.Kill()with|ex->printfn$"Failed to Kill Procees with PID: [{pid}]\n{ex.Message}"// Helper functions to add arguments to the fable commandletprivateaddOutDir(outdir:stringoption)(args:Builders.ArgumentsBuilder)=matchoutdirwith|Someoutdir->args.Add$"-o {outdir}"|None->argsletprivateaddExtension(extension:stringoption)(args:Builders.ArgumentsBuilder)=matchextensionwith|Someextension->args.Add$"-e {extension}"|None->argsletprivateaddWatch(watch:booloption)(args:Builders.ArgumentsBuilder)=matchwatchwith|Sometrue->args.Add$"--watch"|Somefalse|None->args// we can fire up fable either as a background process// or before calling esbuild for productionletfableCmd(isWatch:booloption)=fun(config:FableConfig)->letexecBinName=ifEnv.isWindowsthen"dotnet.exe"else"dotnet"Cli.Wrap(execBinName).WithArguments(funargs->args.Add("fable").Add(defaultArgconfig.project"./src/App.fsproj")|>addWatchisWatch|>addOutDirconfig.outDir|>addExtensionconfig.extension|>ignore)// we don't do a lot, we simply re-direct the stdio to the console.WithStandardErrorPipe(PipeTarget.ToStream(Console.OpenStandardError())).WithStandardOutputPipe(PipeTarget.ToStream(Console.OpenStandardOutput()))letstopFable()=matchactiveFablewith|Somepid->killActiveProcesspid|None->printfn"No active Fable found"letstartFable(getCommand:FableConfigoption->Command)(config:FableConfigoption)=task{// Execute and wait for it to finishletcmdResult=getCommand(config).ExecuteAsync()activeFable<-SomecmdResult.ProcessIdreturn!cmdResult.Task}
Keeping the process ID on memory might not be the best idea and there can be better ways to handle that but at least for now it works just fine.
Calling the startFable function with fable options, will make fable run on the background, this allows us to have fable output JS files that we will be able to serve.
Reload on change
Reloading on change was an interesting feature to do, first of all I needed a file watcher and I have had heard before that the .NET one wasn't really that great, I also needed to communicate with the frontend when something changed in the backend.
For the file watcher, I tried to search for good alternatives, but to be honest in the end I decided to go with the one in the BCL.
I was kind of scared though how would I manage multiple notifications and events without making it a mess? I had No idea... Thankfully FSharp.Control.Reactive was found and is just what I needed. This library allows you to make observables from events and has a bunch of nice utility functions to work with stream like collections if you've used RxJS or RX.NET you will feel at home with it.
letgetFileWatcher(config:WatchConfig)=letwatchers=// monitor a particular list of addresses(defaultArgconfig.directories(["./src"]|>Seq.ofList))|>Seq.map(fundir->// for each address create a file watcherletfsw=newFileSystemWatcher(dir)fsw.IncludeSubdirectories<-truefsw.NotifyFilter<-NotifyFilters.FileName|||NotifyFilters.Sizeletfilters=defaultArgconfig.extensions(Seq.ofList["*.js""*.css""*.ts""*.tsx""*.jsx""*.json"])// ensure you're monitoring all of the// extensions you want to reload on changeforfilterinfiltersdofsw.Filters.Add(filter)// and ensure you will rise events for them :)fsw.EnableRaisingEvents<-truefsw)letsubs=watchers|>Seq.map(funwatcher->// for each watche react to the following events// Renamed// Changed// Deleted// Created[watcher.Renamed// To prevent overflows and weird behaviors// ensure to throttle the events|>Observable.throttle(TimeSpan.FromMilliseconds(400.))|>Observable.map(fune->{oldName=Somee.OldNameChangeType=Renamedname=e.Namepath=e.FullPath})watcher.Changed|>Observable.throttle(TimeSpan.FromMilliseconds(400.))|>Observable.map(fune->{oldName=NoneChangeType=Changedname=e.Namepath=e.FullPath})watcher.Deleted|>Observable.throttle(TimeSpan.FromMilliseconds(400.))|>Observable.map(fune->{oldName=NoneChangeType=Deletedname=e.Namepath=e.FullPath})watcher.Created|>Observable.throttle(TimeSpan.FromMilliseconds(400.))|>Observable.map(fune->{oldName=NoneChangeType=Createdname=e.Namepath=e.FullPath})]// Merge these observables in a single one|>Observable.mergeSeq){newIFileWatcherwithoverride_.Dispose():unit=watchers// when disposing, dispose every watcher you may have around|>Seq.iter(funwatcher->watcher.Dispose())override_.FileChanged:IObservable<FileChangedEvent>=// merge the the merged observables into a single one!!!Observable.mergeSeqsubs}
With this setup you can easily observe changes to multiple directories and multiple extensions it might not be the most efficient way to do it, but It at least got me started with it, now that I had a way to know when something changed I needed to tell the browser what had happened.
For that I chose SSE (Server Sent Events) which is a really cool way to do real time notifications from the server exclusively without having to implement web sockets it's just an HTTP call which can be terminated (or not).
letprivateSse(watchConfig:WatchConfig)next(ctx:HttpContext)=task{letlogger=ctx.GetLogger("Perla:SSE")logger.LogInformation$"LiveReload Client Connected"// set up the correct headersctx.SetHttpHeader("Content-Type","text/event-stream")ctx.SetHttpHeader("Cache-Control","no-cache")ctx.SetStatusCode200// send the first eventletres=ctx.Responsedo!res.WriteAsync($"id:{ctx.Connection.Id}\ndata:{DateTime.Now}\n\n")do!res.Body.FlushAsync()// get the observable of file changesletwatcher=Fs.getFileWatcherwatchConfiglogger.LogInformation$"Watching %A{watchConfig.directories} for changes"letonChangeSub=watcher.FileChanged|>Observable.map(funevent->task{matchPath.GetExtensionevent.namewith|Css->// if the change was on a CSS file send the new contentlet!content=File.ReadAllTextAsyncevent.pathletdata=Json.ToTextMinified({|oldName=event.oldName|>Option.map(funvalue->matchvaluewith|Css->value|_->"")name=event.pathcontent=content|})// CSS HMR was basically free!do!res.WriteAsync$"event:replace-css\ndata:{data}\n\n"return!res.Body.FlushAsync()// if it's any other file well... just reload|Typescript|Javascript|Jsx|Json|Other_->letdata=Json.ToTextMinified({|oldName=event.oldNamename=event.name|})logger.LogInformation$"LiveReload File Changed: {event.name}"do!res.WriteAsync$"event:reload\ndata:{data}\n\n"return!res.Body.FlushAsync()})// ensure the task gets done|>Observable.switchTask|>Observable.subscribeignore// if the client closes the browser// then dispose these resourcesctx.RequestAborted.Register(fun_->watcher.Dispose()onChangeSub.Dispose())|>ignore// keep the connection alivewhiletruedo// TBH there must be a better way to do it// but since this is not critical, it works just finedo!Async.Sleep(TimeSpan.FromSeconds1.)return!text""nextctx}
At this time, I also published about SSE on my blog, I really felt it was a really cool thing and decided to share it with the rest of the world :)
Install dependencies
I was really undecided if I wanted to pursue a webpack alernative because
How can you install dependencies without npm?
Do you really want to do
import { useState } from 'https://cdn.skypack.dev/pin/react@v17.0.1-yH0aYV1FOvoIPeKBbHxg/mode=imports,min/optimized/react.js'
On every damned file? oh no no no, I don't think so... Enter the Import Maps, this feature (along esbuild) was the thing that made me realize it was actually possible to ditch out node/webpack/npm entirely (at least in a local and direct way) instead of doing that ugly import from above, if you can provide a import map with your dependencies the rest should be relatively easy
<script type="importmap">{"imports":{"moment":"https://ga.jspm.io/npm:moment@2.29.1/moment.js","lodash":"https://cdn.skypack.dev/lodash"}}</script><!-- Allows you to do the next --><script type="module">importmomentfrom"moment";importlodashfrom"lodash";</script>
So here I was trying to replicate a version of package.json this ended up implementing the perla.jsonc.lock file which is not precisely a lock file, while the URL's there are certainly the pined and production versions of those packages, it's in reality the import map in disguise, to get that information though I had to investigate how to do it. Once again I decided to study snowpack since it's the only frontend dev tool I know it has this kind of mechanism (remote sources), after some investigation and some PoC's I also stumbled upon JSPM's recently released Import Map Generator which is basically what I wanted to do! Skypack, JSPM and Unpkg offer reliable CDN services for production with all of these investigations and gathered knowledge I went to implement fetching dependencies and "installing" them with the dev server tool.
[<RequireQualifiedAccessAttribute>]moduleinternalHttp=openFlurlopenFlurl.Http[<Literal>]letSKYPACK_CDN="https://cdn.skypack.dev"[<Literal>]letSKYPACK_API="https://api.skypack.dev/v1"[<Literal>]letJSPM_API="https://api.jspm.io/generate"letprivategetSkypackInfo(name:string)(alias:string)=// FsToolkit.ErrorHandling FTWtaskResult{tryletinfo={|lookUp=$"%s{name}"|}let!res=$"{SKYPACK_CDN}/{info.lookUp}".GetAsync()ifres.StatusCode>=400thenreturn!PackageNotFoundException|>ErrorletmutablepinnedUrl=""letmutableimportUrl=""// try to get the pinned URL from the headersletinfo=ifres.Headers.TryGetFirst("x-pinned-url",&pinnedUrl)|>notthen{|infowithpin=None|}else{|infowithpin=SomepinnedUrl|}// and the imports as wellletinfo=ifres.Headers.TryGetFirst("x-import-url",&importUrl)|>notthen{|infowithimport=None|}else{|infowithimport=SomeimportUrl|}return// generate the corresponding import map entry[alias,$"{SKYPACK_CDN}{info.pin |> Option.defaultValue info.lookUp}"],// skypack doesn't handle any import maps so the scopes will always be empty[]with|:?Flurl.Http.FlurlHttpExceptionasex->matchex.StatusCode|>Option.ofNullablewith|Somecodewhencode>=400->return!PackageNotFoundException|>Error|_->()return!ex:>Exception|>Error|ex->return!ex|>Error}letgetJspmInfonamealiassource=taskResult{letqueryParams={|install=[|$"{name}"|]env="browser"provider=// JSPM offer various reliable sources// to get your dependenciesmatchsourcewith|Source.Skypack->"skypack"|Source.Jspm->"jspm"|Source.Jsdelivr->"jsdelivr"|Source.Unpkg->"unpkg"|_->printfn$"Warn: An unknown provider has been specied: [{source}] defaulting to jspm""jspm"|}trylet!res=JSPM_API.SetQueryParams(queryParams).GetJsonAsync<JspmResponse>()letscopes=// F# type serialization hits again!// the JSPM response may include a scope object or not// so try to safely check if it exists or notmatchres.map.scopes:>obj|>Option.ofObjwith|None->Map.empty|Somevalue->value:?>Map<string,Scope>return// generate the corresponding import map// entries as well as the scopesres.map.imports|>Map.toList|>List.map(fun(k,v)->alias,v),scopes|>Map.toListwith|:?Flurl.Http.FlurlHttpExceptionasex->matchex.StatusCode|>Option.ofNullablewith|Somecodewhencode>=400->return!PackageNotFoundException|>Error|_->()return!ex:>Exception|>Error}letgetPackageUrlInfo(name:string)(alias:string)(source:Source)=matchsourcewith|Source.Skypack->getSkypackInfonamealias|_->getJspmInfonamealiassource
This was a relatively low effort to implement but it did require finding a way to gather these resources so they can be mapped to json objects. This approach also allows you yo import different version fo the same package in the same application! that can be useful when you want to migrate dependencies slowly rolling them out.
Production Bundles
Just as Installing dependencies, having a production ready build is critical This is where esbuild finally comes into the picture it is a crucial piece of the puzzle. Esbuild while it's written in go and offers a npm package, it provides a single executable binary which can be used in a lot of platforms and and architectures, it distributes itself through the npm registry so it's about downloading the package in the correct way and just executing it like we did for the fable command.
letesbuildJsCmd(entryPoint:string)(config:BuildConfig)=letdirName=(Path.GetDirectoryNameentryPoint).Split(Path.DirectorySeparatorChar)|>Seq.lastletoutDir=matchconfig.outDirwith|Someoutdir->Path.Combine(outdir,dirName)|>Some|None->Path.Combine("./dist",dirName)|>SomeletexecBin=defaultArgconfig.esBuildPathesbuildExecletfileLoaders=getDefaultLodersconfigCli.Wrap(execBin).WithStandardErrorPipe(PipeTarget.ToStream(Console.OpenStandardError())).WithStandardOutputPipe(PipeTarget.ToStream(Console.OpenStandardOutput()))// CliWrap simply allows us to add arguments to commands very easy.WithArguments(funargs->args.Add(entryPoint)|>addEsExternalsconfig.externals|>addIsBundleconfig.bundle|>addTargetconfig.target|>addDefaultFileLoadersfileLoaders|>addMinifyconfig.minify|>addFormatconfig.format|>addInjectsconfig.injects|>addOutDiroutDir|>ignore)
the CLI API from esbuild is pretty simple to be honest and is really effective when it comes to transpilation the benefits are that it not just transpiles Javascript, it also transpiles typescript, jsx and tsx files. Adding to those features esbuild is blazing fast.
Transpilation on the fly
The dev server not only needs to serve JS content to the browser, often it needs to serve Typescript/JSX/TSX as well, and as we found earlier in the post if you serve static content your options for transforming or manipulating these request are severely limited, so I had to make particular middlewares to enable compiling single files on the fly.
let's check a little bit how these are somewhat laid out on Perla
[<RequireQualifiedAccess>]moduleMiddleware=// this function helps us determine a particular extension is in the request path// if it is we will use one of the middlewares below on the calling site.lettransformPredicate(extensions:stringlist)(ctx:HttpContext)=...letcssImport(mountedDirs:Map<string,string>)(ctx:HttpContext)(next:Func<Task>)=...letjsonImport(mountedDirs:Map<string,string>)(ctx:HttpContext)(next:Func<Task>)=...letjsImport(buildConfig:BuildConfigoption)(mountedDirs:Map<string,string>)(ctx:HttpContext)(next:Func<Task>)=task{letlogger=ctx.GetLogger("Perla Middleware")if// for the moment, we just serve the JS as is and don't process itctx.Request.Path.Value.Contains("~perla~")||ctx.Request.Path.Value.Contains(".js")|>notthenreturn!next.Invoke()elseletpath=ctx.Request.Path.Valuelogger.LogInformation($"Serving {path}")letbaseDir,baseName=// check if we're actually monitoring this directory and this file extensionmountedDirs|>Map.filter(fun_v->String.IsNullOrWhiteSpacev|>not)|>Map.toSeq|>Seq.find(fun(_,v)->path.StartsWith(v))// find the file on diskletfilePath=letfileName=path.Replace($"{baseName}/","",StringComparison.InvariantCulture)Path.Combine(baseDir,fileName)// we will serve javascript regardless of what we find on diskctx.SetContentType"text/javascript"tryifPath.GetExtension(filePath)<>".js"thenreturnfailwith"Not a JS file, Try looking with another extension."// if the file exists on disk// and has a js extension then just send it as is// the browser should be able to interpret itlet!content=File.ReadAllBytesAsync(filePath)do!ctx.WriteBytesAsynccontent:>Taskwith|ex->let!fileData=Esbuild.tryCompileFilefilePathbuildConfigmatchfileDatawith|Ok(stdout,stderr)->ifString.IsNullOrWhiteSpacestderr|>notthen// In the SSE code, we added (later on)// an observer for compilation errors and send a message to the client,// this should trigger an "overlay" on the client sideFs.PublishCompileErrstderrdo!ctx.WriteBytesAsync[||]:>Taskelse// if the file got compiled then just write the file to the body// of the requestletcontent=Encoding.UTF8.GetBytesstdoutdo!ctx.WriteBytesAsynccontent:>Task|Errorerr->// anything else, just send a 500ctx.SetStatusCode500do!ctx.WriteTextAsyncerr.Message:>Task}:>TaskletconfigureTransformMiddleware(config:FdsConfig)(appConfig:IApplicationBuilder)=letserverConfig=defaultArgconfig.devServer(DevServerConfig.DefaultConfig())letmountedDirs=defaultArgserverConfig.mountDirectoriesMap.emptyappConfig.Use(Func<HttpContext,Func<Task>,Task>(jsonImportmountedDirs)).Use(Func<HttpContext,Func<Task>,Task>(cssImportmountedDirs)).Use(Func<HttpContext,Func<Task>,Task>(jsImportconfig.buildmountedDirs))|>ignore
It is a pretty simple module (I want to think) that only has some functions that deal with the content of the files and return any compiled result if neededm otherwise just send the file.
Now... Let's take a look at the magic behind let! fileData = Esbuild.tryCompileFile filePath buildConfig to be honest I didn't really know what I was doing, the main line of thought was just to try and find the content on disk and try the next extension if it didn't work. Hah! well
lettryCompileFilefilepathconfig=taskResult{letconfig=(defaultArgconfig(BuildConfig.DefaultConfig()))// since we're using// FsToolkit.ErrorHandling if the operation fails it will// "early return" meaning it won't continue the success pathlet!res=Fs.tryReadFilefilepathletstrout=StringBuilder()letstrerr=StringBuilder()let(_,loader)=resletcmd=buildSingleFileCmdconfig(strout,strerr)res// execute esbuild on the filedo!(cmd.ExecuteAsync()).Task:>Taskletstrout=strout.ToString()letstrerr=strerr.ToString()letstrout=matchloaderwith|Jsx|Tsx->try// if the file needs injects (e.g automatic "import React from 'react'" in JSX files)letinjects=defaultArgconfig.injects(Seq.empty)|>Seq.mapFile.ReadAllText// add those right hereletinjects=String.Join('\n',injects)$"{injects}\n{strout}"with|ex->printfn$"Perla Serve: failed to inject, {ex.Message}"strout|_->strout// return the compilation results// the transpiled output and the error if anyreturn(strout,strerr)}
Surely thats a lot of things to do for a single file, I'm sure it must be quite slow right? Well... It turns out that .NET and Go are quite quite feaking fast
each request takes around 10-20ms and I'm pretty sure it can be improved once the phase of heavy development settles down and the code base stabilizes a little bit more.
Dev Proxy
This one is pretty new, a dev proxy is somewhat necessary specially when you will host your applications on your own server so you are very likely to have URLs like /api/my-endpoint rather than http://my-api.com/api/my-endpoint it also helps you target different environments with a single configuration change, in this case it was not really complex thanks to @Yaurthek who hinted at me one Yarp implementation of a dev proxy, so I ended up basing my work on that.
The whole idea here is to read a json file with some origin -> target mappings and then just adding a proxy to the server application.
letprivategetHttpClientAndForwarder()=// this socket handler is actually disposable// but since technically I will only use one in the whole application// I won't need to dispose itletsocketsHandler=newSocketsHttpHandler()socketsHandler.UseProxy<-falsesocketsHandler.AllowAutoRedirect<-falsesocketsHandler.AutomaticDecompression<-DecompressionMethods.NonesocketsHandler.UseCookies<-falseletclient=newHttpMessageInvoker(socketsHandler)letreqConfig=ForwarderRequestConfig()reqConfig.ActivityTimeout<-TimeSpan.FromSeconds(100.)client,reqConfigletprivategetProxyHandler(target:string)(httpClient:HttpMessageInvoker)(forwardConfig:ForwarderRequestConfig):Func<HttpContext,IHttpForwarder,Task>=// this is actually using .NET6 Minimal API's from asp.net!lettoFunc(ctx:HttpContext)(forwarder:IHttpForwarder)=task{letlogger=ctx.GetLogger("Perla Proxy")let!error=forwarder.SendAsync(ctx,target,httpClient,forwardConfig)// report the errors to the log as a warning// since we don't need to die if a request failsiferror<>ForwarderError.NonethenleterrorFeat=ctx.GetForwarderErrorFeature()letex=errorFeat.Exceptionlogger.LogWarning($"{ex.Message}")}:>TaskFunc<HttpContext,IHttpForwarder,Task>(toFunc)
And then somewher inside the aspnet application configuration
matchgetProxyConfigwith|SomeproxyConfig->appConfig.UseRouting().UseEndpoints(funendpoints->let(client,reqConfig)=getHttpClientAndForwarder()// for each mapping add the url add an endpointfor(from,target)inproxyConfig|>Map.toSeqdolethandler=getProxyHandlertargetclientreqConfigendpoints.Map(from,handler)|>ignore)|None->appConfig
That's it! At least on my initial testing it seems to work fine, I would need to have some feedback on the feature to know if this is actually working for more complex use cases.
Future and Experimental things
What you have seen so far (and some other minor features) are already inside Perla, they are working and they try to provide you a seamless experience for building Single Page Applications however there are still missing pieces for a complete experience. For example Perla doesn't support Sass or Less at the moment and Sass is a pretty common way to write styles on big frontend projects, we are not able to parse out .Vue files or anything else that is not HTML/CSS/JS/TS/JSX/TSX, We do support HMR for CSS files since that is not a complex mechanism but, HMR for JS/TS/JSX/TSX files is not there yet sady. Fear not that We're looking for a way to provide these at some point in time.
Plugins
I'm a fan of .fsx files, F# scripts are pretty flexible and since F# 5.0 they are even more powerful than ever allowing you to pull dependencies directly from NuGet without any extra command.
The main goal for the author and user experiences is somewhat like this
As an Author:
Write an .fsx script
Upload it to gist/github
Profit
As a User:
Add an entry yo tour "plugins" section
Profit
Implementation details are more complex though...
My vision is somewhere along the following lines
get a request for a different file e.g. sass files
if the file is not part of the default supported extensions
Call a function that will parse the content of that file
get the transpiled content or the compilation error (just like we saw above with the js middleware)
return the valid HTML/CSS/JS content to the browser
To get there I want to leverage the [FSharp.Compiler.Services] NuGet Package to start an F# interactive session that runs over the life of the server,
Start the server, also if there are plugins in the plugin section, start the fsi session.
load the plugins, download them to a known location in disk, or even just get the strings without downloading the file to disk
execute the contents on the fsi session and grab a particular set of functions
These functions can be part of a life cycle which may possible be something like
on load // when HMR is enabled
on change // when HMR is enabled
on transform // when the file is requested
on build // when the production build is executed
call the functions in the plugins section whenever needed
Starting an FSI session is not a complex task let's take a look.
In this file we're able to provide a function that when given a file path, it will try to compile a .scss file into it's .css equivalent, to be able to execute that in Perla, we need a module that does somewhat like this:
#r"nuget: FSharp.Compiler.Service, 41.0.1"openSystemopenSystem.IOopenFSharp.Compiler.Interactive.ShellmoduleScriptedContent=lettryGetSassPluginFunction(content:string):(string->string)option=letdefConfig=FsiEvaluationSession.GetDefaultConfiguration()letargv=[|"fsi.exe""--noninteractive""--nologo""--gui-"|]usestdIn=newStringReader("")usestdOut=newStringWriter()usestdErr=newStringWriter()usesession=FsiEvaluationSession.Create(defConfig,argv,stdIn,stdOut,stdErr,true)session.EvalInteractionNonThrowing(content)|>ignorematchsession.TryFindBoundValue"compileSassFile"with|Somebound->// If there's a value with that name on the script try to grab itmatchbound.Value.ReflectionValuewith// ensure it fits the signature we are expecting|:?FSharpFunc<string,string>ascompileSassFile->SomecompileSassFile|_->None|None->Noneletcontent=File.ReadAllText("./path/to/sass-plugin.fsx")// this is where it get's nice, we can also fetch the scritps from the cloud// let! content = Http.getFromGithub("AngelMunoz/Perla.Sass")matchScriptedContent.tryGetSassPluginFunction(content)with|Someplugin->letcss=plugin"./path/to/file.scss"printfn$"Resulting CSS:\n{css}"|None->printfn"No plugin was found on the script"
This is more-less what I have in mind, it has a few downsides though
Convention based naming
Badly written plugins might leak memory or make Perla's performance to slow down
Script distribution is a real concern, there's no clear way to do it as of now
Security concerns when executing code with Perla's permissions on the user's behalf
And many others that I might not be looking after.
Being able to author plugins and process any kind of file into something Perla can use to enhance the consumer experience is just worth it though, for example just look at the vast amount of webpack and vite plugins. The use cases are there for anyone to fulfill them .
HMR
This is the golden apple I'm not entirely sure how to tackle... There's an HMR spec that I will follow for that since that's what snowpack/vite's HMR is based on, libraries like Fable.Lit, or Elmish.HMR are working towards being compatible with vite's HMR, so if Perla can make it work like them, then we won't even need to write any specific code for Perla.
I can talk however of CSS HMR, This is a pretty simple change to support given that CSS changes are automatically propagated in the browser, it basically does half of the HMR for us.
Perla does the following:
Sees import "./app.css
Runs the cssImport middleware function I hinted at earlier and returns a ~CSS~ Javascript file that injects a script tag on the head of the page.
letcssImport(mountedDirs:Map<string,string>)(ctx:HttpContext)(next:Func<Task>)=task{// skip non-css filesifctx.Request.Path.Value.Contains(".css")|>notthenreturn!next.Invoke()elseletlogger=ctx.GetLogger("Perla Middleware")letpath=ctx.Request.Path.ValueletbaseDir,baseName=mountedDirs|>Map.filter(fun_v->String.IsNullOrWhiteSpacev|>not)|>Map.toSeq|>Seq.find(fun(_,v)->path.StartsWith(v))letfilePath=letfileName=path.Replace($"{baseName}/","",StringComparison.InvariantCulture)Path.Combine(baseDir,fileName)logger.LogInformation("Transforming CSS")let!content=File.ReadAllTextAsync(filePath)// return the JS code to insert the CSS content in a style tagletnewContent=$"""
const css = `{content}`
const style = document.createElement('style')
style.innerHTML = css
style.setAttribute("filename", "{filePath}");
document.head.appendChild(style)"""ctx.SetContentType"text/javascript"do!ctx.WriteStringAsyncnewContent:>Task}:>Task
In the SSE handler function we observe for file changes in disk and depending on the content we do the corresponding update
watcher.FileChanged|>Observable.map(funevent->task{matchPath.GetExtensionevent.namewith|Css->// a CSS file was changed, read all of the contentlet!content=File.ReadAllTextAsyncevent.pathletdata=Json.ToTextMinified({|oldName=event.oldName|>Option.map(funvalue->matchvaluewith|Css->value|_->"")name=event.pathcontent=content|})// Send the SSE Message to the client with the new CSS contentdo!res.WriteAsync$"event:replace-css\ndata:{data}\n\n"return!res.Body.FlushAsync()|Typescript|Javascript|Jsx|Json|Other_->//... other content ...})|>Observable.switchTask|>Observable.subscribeignore
To handle these updates we use two cool things, WebWorkers and a simple scripts, the live reload script has this content
// initiate workerconstworker=newWorker("/~perla~/worker.js");// connect to the SSE endpointworker.postMessage({event:"connect"});functionreplaceCssContent({oldName,name,content}){constcss=content?.replace(/(?:\\r\\n|\\r|\\n)/g,"\n")||"";constfindBy=oldName||name;// find the style tag with the particular nameconststyle=document.querySelector(`[filename="${findBy}"]`);if (!style){console.warn("Unable to find",oldName,name);return;}// replace the contentstyle.innerHTML=css;style.setAttribute("filename",name);}functionshowOverlay({error}){console.log("show overlay");}// react to the worker messagesworker.addEventListener("message",function ({data}){switch (data?.event){case"reload":returnwindow.location.reload();case"replace-css":returnreplaceCssContent(data);case"compile-err":returnshowOverlay(data);default:returnconsole.log("Unknown message:",data);}});
Inside our Worker the code is very very similar
letsource;consttryParse=(string)=>{try{returnJSON.parse(string)||{};}catch (err){return{};}};functionconnectToSource(){if (source)return;//connect to the SSE endpointsource=newEventSource("/~perla~/sse");source.addEventListener("open",function (event){console.log("Connected");});// react to file reloadssource.addEventListener("reload",function (event){console.log("Reloading, file changed: ",event.data);self.postMessage({event:"reload",});});// if the server sends a `replace-css` event// notify the main thread about it// Yes! web workers run on background threads!source.addEventListener("replace-css",function (event){const{oldName,name,content}=tryParse(event.data);console.log(`Css Changed: ${oldName?oldName:name}`);self.postMessage({event:"replace-css",oldName,name,content,});});source.addEventListener("compile-err",function (event){const{error}=tryParse(event.data);console.error(error);self.postMessage({event:"compile-err",error,});});}self.addEventListener("message",function ({data}){if (data?.event==="connect"){connectToSource();}});
And that's how the CSS HMR works in Perla and it is instant, in less than a blink of an eye! Well... maybe not but pretty close to it.
For the JS side I'm still not sure how this will work given that I might need to have a mapping in both sides of the files I have and what is their current version.
What's next?
Whew! That was a lot! but shows how to build each part of the Webpack alternative I've been working on Called Perla there are still some gaps though
Project Scaffolding
This will be an important step for adoption I believe, generating certain files, or even starter projects to reduce the onboarding complexity is vital, so this is likely the next step for me (even before the HMR)
Unit/E2E Testing
Node based test runners won't work naturally since we're not using node! So this is an area to investigate, for E2E I already have the thought of using playwright, for unit tests I'm not sure yet but I guess I'd be able to pick something similar or simply have a test framework that runs entirely on the browser.
Import map ergonomics
Some times, you must edit by hand the import map (perla.jsonc.lock) to get dependencies like import fp from 'lodash/fp' with the import maps the browser knows what to do with lodash but not lodash/fp so an edit must be made, this requires you to understand how these dependencies work and how you need to write the import map, it's an area I'd love to make as simple as possible
Typescript Types for dependencies
Typescript (and related Editors/IDEs like vscode and webstorm) rely on the presence of node_modules to pick typings from disk, It would be nice if typescript worked with the URI style imports for typings that would fix a lot of issues.
Library/Build only mode
There might be certain cases where you would like to author a library either for you and your team, perhaps you only need the JS files and a package.json to share the sources, while not a priority it's something it's worth looking at.
Improve the Install story
The goal is run a single line command, get your stuff installed in place regardless of if you have .NET or not
Closing thoughts...
So those are some of the things I might have on my plate for next, of course if I receive feedback on this project I may prioritize some things over the others but rather than doing it all for myself, I wish to share this and make it a community effort to continue to improve the Frontend tooling story that doesn't rely on complex patterns and extremely weird and cryptic errors, hundreds of hours spent in configuration, something as simple that you feel confident enough to trust and use :)
A cross-platform tool for unbundled front-end development that doesn't depend on Node or requires you to install a complex toolchain
Perla Dev Server
Perla is a cross-platform single executable binary CLI Tool for a Development Server of Single Page Applications (like vite/webpack but no node required!).
Description
The Perla Dev Server!
Usage:
Perla [command] [options]
Options:
--version Show version information
-?, -h, --help Show help and usage information
--info <info> Brings the Help dialog []
Commands:
setup Initialized a given directory or perla itself
t, templates <templateRepositoryName> Handles Template Repository operations such as list, add, update, and remove templates
describe, ds <properties> Describes the perla.json file or it's properties as requested
b, build Builds the SPA application for distribution
s, serve, start Starts the development server and if fable projects are present it also takes care of it.
add, install <package> Shows information about a package if the name matches an existing one
remove <package> removes a package from the
list,
And with all due respect, I really thank the maintainers of snowpack,esbuild, and vite who make an incredible job at reducing the complexity of frontend tooling as well, they inspired me AND if you're a loving node user please look at their projects ditch webpack enough so they also reflect back and simplify their setups!
For the .NET community I wish to spark a little of interest to look outside and build tooling for other communities, I think it is a really nice way to introduce .NET with the rest of the world and new devs be it with F#, C# or VB, I think .NET is an amazing platform for that. This has been a really good learning exercise and at the same time a project I believe in so, I will try to spend more time and push it to as much as I can.