WIP: Adds basic support for LiveReload when watching/previewing

[Silvenga]

This pull request adds basic support for the LiveReload protocol when watching and previewing. Created as I use (https://github.com/gruntjs/grunt-contrib-watch#optionslivereload) religiously and due to my chronic lazyness problem (I'm getting help).

LiveReload - LiveReload is a tool for web developers and designers that automatically refreshes connected browsers. Some changes like CSS modifications can be partially reloaded without refreshing the whole page.

http://feedback.livereload.com/knowledgebase/articles/86174-livereload-protocol
https://github.com/livereload/livereload-js

TODO

• Owin should host livereload.js directly (required by the LiveReload protocol).
  • Right now it requires the script to be in the output directory. This is to support browser extensions without needing to include livereload.js in the site source. e.g. https://chrome.google.com/webstore/detail/livereload/jnihajbhpnppcggbcgedagnkighmdlei?hl=en
  • The script could be embedded in the assembly, should it be overriden by the user's version?
• Support for just watching (not requiring preview running).
• Add documentation.
• Add support for alerts.
• Module to inject in LiveReload without extension.
  • To support multiple LiveReload servers coexisting.
  • This should work:

new HtmlInsert("body", "<script async src='livereload.js'></script>")

• Some unit tests.

Questions

• Is the protocol implemented sufficiently?
• Is ReloadClientServiceLocator implemented sufficiently?

[daveaglick]

This is really awesome, thanks! I assume that it's a work in progress from the todo list? Weekends are typically family time so I don't do much OSS, but I'll review this early next week and get you comments if I have any.

With regard to unit tests - I'm not totally strict about everything needing to have tests. In general I prefer to test component integration or anything that may break in the future due to refactoring. This is probably worth at least a couple tests, for example to make sure the engine is sending the correct signals. I'll get you better feedback once I've had a chance to take a look.

Thanks again!

[Silvenga]

Rebased onto develop.

Yep, a WIP. I've been using this pull to develop my blog and am happy with the current LiveReload support. I want to move onto hardening the code somewhat now - going to look into adding some test coverage now.

Questions:
How do you want me to document this - if any?

[daveaglick]

I wouldn't worry too much about documentation. The easiest thing is probably just to add another paragraph to https://wyam.io/docs/usage/command-line#embedded-web-server. I think you can remove that from the todo list here - it's easy enough to write a little something up after the fact.

I also noticed some tests, which is great. I'm going to try and get a release out later today or early tomorrow. I would love to include this in there. I'm going to bring down the PR and try it out locally, but assuming I don't hit any bumps are there any reasons why this isn't ready to roll?

[daveaglick]

So I'm running into a little trouble. I'm hitting a System.Net.HttpListenerException, "Access is denied" in LiveReloadServer here:

public void StartStandaloneHost(int port = 35729)
{
    try
    {
        // This must be 127.0.0.1 due to http.sys hostname verification, LiveReload hardcodes it.
        StartOptions options = new StartOptions("http://127.0.0.1:" + port);
        options.Settings.Add(typeof(ITraceOutputFactory).FullName, typeof(NullTraceOutputFactory).AssemblyQualifiedName);
        _server = WebApp.Start(options, InjectOwinMiddleware);
    }
    catch (Exception ex)
    {
        Trace.Warning($"Error while running the LiveReload server: {ex.Message}");
    }

    Trace.Verbose($"LiveReload server listening on port {port}.");
}

Suspect it's because we're trying to bind to a specific IP address. Also found this issue from SignalR that suggests there's no good way around it other than running the application as an admin.

I noticed you have the comment "This must be 127.0.0.1 due to http.sys hostname verification" - can you expand on that a little? It wouldn't work if we localhost as the preview server does?

[daveaglick]

Some additional brainstorming - what do you think about the preview server doing some simple HTML injection for every HTML request?

I.e., if LiveReload is active, inject the <script async src='livereload.js'></script> just before the closing </body> element on every request before sending the HTML out. That way we wouldn't be sticking anything about livereload.js into the output.

[daveaglick]

I'm seeing the following network error in Chrome (happens whether I use localhost or 127.0.0.1):

WebSocket connection to 'ws://127.0.0.1:35729/livereload' failed: Error during WebSocket handshake: Unexpected response code: 400

[Silvenga]

I noticed you have the comment "This must be 127.0.0.1 due to http.sys hostname verification" - can you expand on that a little? It wouldn't work if we localhost as the preview server does?

If localhost is set under Owin, Owin requires that every connecting client have Host: localhost in their headers. I was annoying getting a 400 response whenever a client using 127.0.0.1 connected. IIRC this verification is done by http.sys, and I'm not sure how to override this under Owin.

Access is denied

That's odd. I'll bring up a VM and try to reproduce. I'm not running under admin rights.

what do you think about the preview server doing some simple HTML injection for every HTML request?

Love the idea! I was preparing a module for that, however, I didn't want to duplicate code with https://wyam.io/modules/htmlinsert. What do you think is the best way to handle that?

I guess this is also where the browser extensions come into play.

I'm seeing the following network error in Chrome

When running with verbose logging, what do you get?

Do you have anything else listening on port 35729? I think that would explain all your problems. EDIT: Wouldn't make sense.

[Silvenga]

Got a VM working, steps I went through:

• Got a Windows 10 VM from https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/
• Forgot the login for the admin user
• Created a standard user account
• Enabled remote login (for HyperV)
• Logged in, copied Wyam compiled from my branch
• Broke my bluetooth drivers when creating a switch for HyperV
• Copied my blog's source
• Desktop crashed due to bluetooth drivers
• Ran using wyam -v -w -p
• Check Livereload with Edge using the script tag (uses port 5080) - checked network + console
• Installed Chrome + Chrome LiveReload extension (uses port 35729) - checked network + console

I couldn't reproduce any issues. 😿

[Silvenga]

Oh, crap, are you running Windows 7? Forgot that Websockets (from System.Net) are not supported by Microsoft under anything less than Windows 8. https://msdn.microsoft.com/en-us/library/system.net.websockets(v=vs.110).aspx

This is due to an issue with http.sys.

The library I'm using turns out to use System.Net.Websockets - Owin.Websockets. I looked for others but Livereload requires that the server serve the files + the socket on the same port. This would be impossible with Owin running.

EDIT: We need this: bryceg/Owin.WebSocket#20

[Silvenga]

I'll look into getting a Windows 7 compatible library to work nicely with Owin. The upgrade connection code for Owin is rather simple, I might just fork the two above libraries.

[Silvenga]

Rebased with develop - now using Fleck.

@daveaglick I believe this should work now under Windows 7. I'm going to have to find a Windows 7 VM somewhere to test it properly...

[daveaglick]

Thanks for the continued effort on this one. My main dev box is Win 7 so I should be able to give this a go in the next couple days.

[daveaglick]

Okay, taking a look at this now...

With regard to code injection:

I was preparing a module for that, however, I didn't want to duplicate code with https://wyam.io/modules/htmlinsert. What do you think is the best way to handle that?

My opinion is that it shouldn't be in a module at all. If the injection is done through a module than that means that the configuration (or recipe) would need to know about it. LiveReload should be totally outside the scope of the configuration IMO and should be a lower-level concern. My thinking was that an OWIN middleware would do some simple text scanning for </head> and/or </body> and inject whatever needs to be inserted just before that for HTML requests when LiveReload is turned on.

[daveaglick]

Looked briefly at Fleck - looks like a good choice. While it's not .NET Core yet, it does look like it's in the works in statianzo/Fleck#175 (last activity 10 days ago). That's important since Wyam will be going .NET Core before too long as well.

[Silvenga]

inject whatever needs to be inserted just before that for HTML requests when LiveReload is turned on.

Oh, didn't think about that, basically need to create some middleware that runs after static file requests that detects content type (.html or .htm) and injects in the LR scripts at the end of the <body/> node. Might be a little slow, but it's a dev server anyway.

I'm thinking it should be best effort - if the HTML is malformed or missing a body tag, give up. This means we could easily just use built-in XML parsing to reduce our dependencies. I'll look into the middleware when I have some time, maybe today.

Do you want this feature in this pull, or another down the road?

Looked briefly at Fleck - looks like a good choice.

Unfortunately had to fork the Owin middleware (https://github.com/Silvenga/Owin.WebSocket) and had to publish Owin.WebSocket.Fleck with Owin.WebSocket bundled as the nuget.org version was too old. Kind of a hack - I'll look into upgrading Fleck when dotNetCore version is stabilized.

[daveaglick]

Do you want this feature in this pull, or another down the road?

Don't worry about it right now - I'll probably do a little hacking and add this myself right after merge.

Made some progress today - still getting some web socket errors on my client, but I think I'm getting close to figuring out why. I'm still alternating between access denied errors while running the LiveReload server and WebSocket connection errors on the client. I am gaining a much better appreciation for the mechanics of what's going on though, so the exercise has been helpful.

Going to need to stop for the day soon, but I'll pick it up again tomorrow. I'm hoping to get stuff straightened out on my side so I can merge this in soon. It's a great feature.

[Silvenga]

still getting some web socket errors on my client

Well then, I need to find a Windows 7 VM.

[Silvenga]

Added testing around hostname verifications. Windows 7 should fail to run the tests though...

@daveaglick it only happens under <= Windows 7 from what I can tell. Windows 7 needs URL Reservations (http://stackoverflow.com/questions/24976425/running-self-hosted-owin-web-api-under-non-admin-account) for everything except localhost. Windows 7 does not believe 127.0.0.1 or [::1] are localhost.

Windows 8+ does seem count 127.0.0.1 as localhost however - which is why it works on my development environment as well as Appveyor, but not on yours.

The test ServerShouldAcceptRequestsFrom127001 shows why I need to bind to 127.0.0.1.

[Silvenga]

Root cause - It doesn't look like the required delegates are being registered by Katana with OWIN. This means that no websockets can take place. http://owin.org/spec/extensions/owin-WebSocket-Extension-v0.4.0.htm

This is painful... I wonder if upgrading to Kestrel would help (or be practical).

[Silvenga]

If it means anything, got Kestrel to work under Windows 7 with Websockets (I only upgraded the LR server)...

It should inplace upgrade from using Kantana on the preview server (as Owin compatibility is kept). It might even be prefered considering the move to dotNetCore. This would completely move this project away from http.sys and bring compatibility to Mono. My current branch is full of hacks (not required in the end) and is a mess, I'll cleanup before committing.

I did however have issues with VS2015 not playing nice with libuv - not getting copied into the output directory. It might work in VS2017... I'll test it tomorrow.

[Silvenga]

Rebased onto develop with #449. Everything looks good under W10/W7 - interface binding should no longer be an issue and sockets just work. This also includes the injection middleware with some test coverage.

[daveaglick]

Yay! I'll take a look first thing in the morning and hopefully get it merged ASAP now.

[daveaglick]

Wow. Just wow. It works and it's totally amazing. Going to merge in a minute. Fantastic work, and thanks for sticking with it while we worked out all the bumps.