Update version to 0.1-rc.4

README.md

@@ -9,6 +9,7 @@ Ultimate camera streaming application with support RTSP, WebRTC, HomeKit, FFmpeg
 - streaming from [RTSP](#source-rtsp), [RTMP](#source-rtmp), [MJPEG](#source-ffmpeg), [HLS/HTTP](#source-ffmpeg), [USB Cameras](#source-ffmpeg-device) and [other sources](#module-streams)
 - streaming to [RTSP](#module-rtsp), [WebRTC](#module-webrtc), [MSE/MP4](#module-mp4) or [MJPEG](#module-mjpeg)
 - first project in the World with support streaming from [HomeKit Cameras](#source-homekit)
+- first project in the World with support H265 for WebRTC in browser (only Safari)
 - on the fly transcoding for unsupported codecs via [FFmpeg](#source-ffmpeg)
 - multi-source 2-way [codecs negotiation](#codecs-negotiation)
    - mixing tracks from different sources to single stream
@@ -26,35 +27,6 @@ Ultimate camera streaming application with support RTSP, WebRTC, HomeKit, FFmpeg
 - [MediaSoup](https://mediasoup.org/) framework routing idea
 - HomeKit Accessory Protocol from [@brutella](https://github.com/brutella/hap)
 
-## Codecs negotiation
-
-For example, you want to watch RTSP-stream from [Dahua IPC-K42](https://www.dahuasecurity.com/fr/products/All-Products/Network-Cameras/Wireless-Series/Wi-Fi-Series/4MP/IPC-K42) camera in your Chrome browser.
-
-- this camera support 2-way audio standard **ONVIF Profile T**
-- this camera support codecs **H264, H265** for send video, and you select `H264` in camera settings
-- this camera support codecs **AAC, PCMU, PCMA** for send audio (from mic), and you select `AAC/16000` in camera settings
-- this camera support codecs **AAC, PCMU, PCMA** for receive audio (to speaker), you don't need to select them
-- your browser support codecs **H264, VP8, VP9, AV1** for receive video, you don't need to select them
-- your browser support codecs **OPUS, PCMU, PCMA** for send and receive audio, you don't need to select them
-- you can't get camera audio directly, because its audio codecs doesn't match with your browser codecs 
-   - so you decide to use transcoding via FFmpeg and add this setting to config YAML file
-   - you have chosen `OPUS/48000/2` codec, because it is higher quality than the `PCMU/8000` or `PCMA/8000`
-
-Now you have stream with two sources - **RTSP and FFmpeg**:
-
-```yaml
-streams:
-  dahua:
-    - rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
-    - ffmpeg:rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0#audio=opus
-```
-
-**go2rtc** automatically match codecs for you browser and all your stream sources. This called **multi-source 2-way codecs negotiation**. And this is one of the main features of this app.
-
-![](assets/codecs.svg)
-
-**PS.** You can select `PCMU` or `PCMA` codec in camera setting and don't use transcoding at all. Or you can select `AAC` codec for main stream and `PCMU` codec for second stream and add both RTSP to YAML config, this also will work fine.
-
 ## Fast start
 
 1. Download [binary](#go2rtc-binary) or use [Docker](#go2rtc-docker) or [Home Assistant Add-on](#go2rtc-home-assistant-add-on)
@@ -207,24 +179,27 @@ streams:
   hls: ffmpeg:https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/gear5/prog_index.m3u8#video=copy
 
   # [MJPEG] video will be transcoded to H264
-  mjpeg: ffmpeg:http://185.97.122.128/cgi-bin/faststream.jpg?stream=half&fps=15#video=h264
+  mjpeg: ffmpeg:http://185.97.122.128/cgi-bin/faststream.jpg#video=h264
 
   # [RTSP] video with rotation, should be transcoded, so select H264
-  rotate: ffmpeg:rtsp://rtsp:12345678@192.168.1.123/av_stream/ch0#raw=-vf transpose=1#video=h264
+  rotate: ffmpeg:rtsp://rtsp:12345678@192.168.1.123/av_stream/ch0#video=h264#rotate=90
 ```
 
-All trascoding formats has [built-in templates](https://github.com/AlexxIT/go2rtc/blob/master/cmd/ffmpeg/ffmpeg.go): `h264`, `h264/ultra`, `h264/high`, `h265`, `opus`, `pcmu`, `pcmu/16000`, `pcmu/48000`, `pcma`, `pcma/16000`, `pcma/48000`, `aac/16000`.
+All trascoding formats has [built-in templates](https://github.com/AlexxIT/go2rtc/blob/master/cmd/ffmpeg/ffmpeg.go): `h264`, `h264/ultra`, `h264/high`, `h265`, `opus`, `pcmu`, `pcmu/16000`, `pcmu/48000`, `pcma`, `pcma/16000`, `pcma/48000`, `aac`, `aac/16000`.
 
 But you can override them via YAML config. You can also add your own formats to config and use them with source params.
 
 ```yaml
 ffmpeg:
   bin: ffmpeg                                        # path to ffmpeg binary
-  h264: "-codec:v libx264 -g 30 -preset superfast -tune zerolatency -profile main -level 4.1"
+  h264: "-codec:v libx264 -g:v 30 -preset:v superfast -tune:v zerolatency -profile:v main -level:v 4.1"
   mycodec: "-any args that support ffmpeg..."
 ```
 
-Also you can use `raw` param for any additional FFmpeg arguments. As example for video rotation (`#raw=-vf transpose=1`). Remember that rotation is not possible without transcoding, so add supported codec as second param (`#video=h264`).
+- You can use `video` and `audio` params multiple times (ex. `#video=copy#audio=copy#audio=pcmu`)
+- You can use go2rtc stream name as ffmpeg input (ex. `ffmpeg:camera1#video=h264`)
+- You can use `rotate` params with `90`, `180`, `270` or `-90` values, important with transcoding (ex. `#video=h264#rotate=90`)
+- You can use `raw` param for any additional FFmpeg arguments (ex. `#raw=-vf transpose=1`).
 
 #### Source: FFmpeg Device
 
@@ -279,6 +254,24 @@ You can pair device with go2rtc on the HomeKit page. If you can't see your devic
 
 If you see a device but it does not have a pair button - it is paired to some ecosystem (Apple Home, Home Assistant, HomeBridge etc). You need to delete device from that ecosystem, and it will be available for pairing. If you cannot unpair device, you will have to reset it.
 
+**Important:**
+
+- HomeKit audio uses very non-standard **AAC-ELD** codec with very non-standard params and specification violation
+- Audio can be transcoded by [ffmpeg](#source-ffmpeg) source with `#async` option
+- Audio can be played by `ffplay` with `-use_wallclock_as_timestamps 1 -async 1` options
+- Audio can't be played in `VLC` and probably any other player
+
+Recommended settings for using HomeKit Camera with WebRTC, MSE, MP4, RTSP:
+
+```
+streams:
+  aqara_g3:
+    - hass:Camera-Hub-G3-AB12
+    - ffmpeg:aqara_g3#audio=aac#audio=opus#async
+```
+
+RTSP link with "normal" audio for any player: `rtsp://192.168.1.123:8554/aqara_g3?video&audio=aac`
+
 **This source is in active development!** Tested only with [Aqara Camera Hub G3](https://www.aqara.com/eu/product/camera-hub-g3) (both EU and CN versions).
 
 #### Source: Ivideon
@@ -320,9 +313,10 @@ The HTTP API is the main part for interacting with the application. Default addr
 
 ```yaml
 api:
-  listen: ":1984"  # HTTP API port ("" - disabled)
+  listen: ":1984"    # HTTP API port ("" - disabled)
-  base_path: ""    # API prefix for serve on suburl
+  base_path: "/rtc"  # API prefix for serve on suburl (/api => /rtc/api)
-  static_dir: ""   # folder for static files (custom web interface)
+  static_dir: "www"  # folder for static files (custom web interface)
+  origin: "*"        # allow CORS requests (only * supported)
 ```
 
 **PS. go2rtc** doesn't provide HTTPS or password protection. Use [Nginx](https://nginx.org/) or [Ngrok](#module-ngrok) or [Home Assistant Add-on](#go2rtc-home-assistant-add-on) for this tasks.
@@ -557,18 +551,17 @@ PS. Additionally WebRTC opens a lot of random UDP ports for transmit encrypted m
 
 Device | WebRTC | MSE | MP4
 -------|--------|-----|----
-*latency*       | best        | medium      | bad
+*latency*           | best        | medium      | bad
-Desktop Chrome  | H264        | H264, H265* | H264, H265*
+Desktop Chrome 107+ | H264        | H264, H265* | H264, H265*
-Desktop Safari  | H264, H265* | H264        | no
+Desktop Safari      | H264, H265* | H264, H265  | **no!**
-Desktop Edge    | H264        | H264, H265* | H264, H265*
+Desktop Edge        | H264        | H264, H265* | H264, H265*
-Desktop Firefox | H264        | H264        | H264
+Desktop Firefox     | H264        | H264        | H264
-Desktop Opera   | no          | H264        | H264
+iPad Safari 13+     | H264, H265* | H264, H265  | **no!**
-iPhone Safari   | H264, H265* | no          | no
+iPhone Safari 13+   | H264, H265* | **no!**     | **no!**
-iPad Safari     | H264, H265* | H264        | no
+Android Chrome 107+ | H264        | H264, H265* | H264
-Android Chrome  | H264        | H264        | H264
+masOS Hass App      | no          | no          | no
-masOS Hass App  | no          | no          | no
 
-- Chrome H265: [read this](https://github.com/StaZhu/enable-chromium-hevc-hardware-decoding)
+- Chrome H265: [read this](https://chromestatus.com/feature/5186511939567616) and [read this](https://github.com/StaZhu/enable-chromium-hevc-hardware-decoding)
 - Edge H265: [read this](https://www.reddit.com/r/MicrosoftEdge/comments/v9iw8k/enable_hevc_support_in_edge/)
 - Desktop Safari H265: Menu > Develop > Experimental > WebRTC H265
 - iOS Safari H265: Settings > Safari > Advanced > Experimental > WebRTC H265
@@ -578,6 +571,41 @@ masOS Hass App  | no          | no          | no
 - WebRTC audio codecs: `PCMU/8000`, `PCMA/8000`, `OPUS/48000/2`
 - MSE/MP4 audio codecs: `AAC`
 
+**Apple devices**
+
+- all Apple devices don't support MP4 stream (they only support progressive loading of static files)
+- iPhones don't support MSE technology because it competes with the HLS technology, invented by Apple
+- HLS is the worst technology for **live** streaming, it still exists only because of iPhones
+
+## Codecs negotiation
+
+For example, you want to watch RTSP-stream from [Dahua IPC-K42](https://www.dahuasecurity.com/fr/products/All-Products/Network-Cameras/Wireless-Series/Wi-Fi-Series/4MP/IPC-K42) camera in your Chrome browser.
+
+- this camera support 2-way audio standard **ONVIF Profile T**
+- this camera support codecs **H264, H265** for send video, and you select `H264` in camera settings
+- this camera support codecs **AAC, PCMU, PCMA** for send audio (from mic), and you select `AAC/16000` in camera settings
+- this camera support codecs **AAC, PCMU, PCMA** for receive audio (to speaker), you don't need to select them
+- your browser support codecs **H264, VP8, VP9, AV1** for receive video, you don't need to select them
+- your browser support codecs **OPUS, PCMU, PCMA** for send and receive audio, you don't need to select them
+- you can't get camera audio directly, because its audio codecs doesn't match with your browser codecs
+  - so you decide to use transcoding via FFmpeg and add this setting to config YAML file
+  - you have chosen `OPUS/48000/2` codec, because it is higher quality than the `PCMU/8000` or `PCMA/8000`
+
+Now you have stream with two sources - **RTSP and FFmpeg**:
+
+```yaml
+streams:
+  dahua:
+    - rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
+    - ffmpeg:rtsp://admin:password@192.168.1.123/cam/realmonitor?channel=1&subtype=0#audio=opus
+```
+
+**go2rtc** automatically match codecs for you browser and all your stream sources. This called **multi-source 2-way codecs negotiation**. And this is one of the main features of this app.
+
+![](assets/codecs.svg)
+
+**PS.** You can select `PCMU` or `PCMA` codec in camera setting and don't use transcoding at all. Or you can select `AAC` codec for main stream and `PCMU` codec for second stream and add both RTSP to YAML config, this also will work fine.
+
 ## TIPS
 
 **Using apps for low RTSP delay**

cmd/api/api.go

@@ -17,6 +17,7 @@ func Init() {
 			Listen    string `yaml:"listen"`
 			BasePath  string `yaml:"base_path"`
 			StaticDir string `yaml:"static_dir"`
+			Origin    string `yaml:"origin"`
 		} `yaml:"api"`
 	}
 
@@ -34,7 +35,7 @@ func Init() {
 	log = app.GetLogger("api")
 
 	initStatic(cfg.Mod.StaticDir)
-	initWS()
+	initWS(cfg.Mod.Origin)
 
 	HandleFunc("api/streams", streamsHandler)
 	HandleFunc("api/ws", apiWS)
@@ -48,16 +49,18 @@ func Init() {
 
 	log.Info().Str("addr", cfg.Mod.Listen).Msg("[api] listen")
 
+	s := http.Server{}
+	s.Handler = http.DefaultServeMux
+
+	if log.Trace().Enabled() {
+		s.Handler = middlewareLog(s.Handler)
+	}
+
+	if cfg.Mod.Origin == "*" {
+		s.Handler = middlewareCORS(s.Handler)
+	}
+
 	go func() {
-		s := http.Server{}
-
-		if log.Trace().Enabled() {
-			s.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-				log.Trace().Stringer("url", r.URL).Msgf("[api] %s", r.Method)
-				http.DefaultServeMux.ServeHTTP(w, r)
-			})
-		}
-
 		if err = s.Serve(listener); err != nil {
 			log.Fatal().Err(err).Msg("[api] serve")
 		}
@@ -83,6 +86,21 @@ var basePath string
 var log zerolog.Logger
 var wsHandlers = make(map[string]WSHandler)
 
+func middlewareLog(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		log.Trace().Msgf("[api] %s %s", r.Method, r.URL)
+		next.ServeHTTP(w, r)
+	})
+}
+
+func middlewareCORS(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Access-Control-Allow-Origin", "*")
+		w.Header().Set("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS")
+		next.ServeHTTP(w, r)
+	})
+}
+
 func streamsHandler(w http.ResponseWriter, r *http.Request) {
 	src := r.URL.Query().Get("src")
 	name := r.URL.Query().Get("name")

cmd/api/ws.go

@@ -4,35 +4,19 @@ import (
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
 	"github.com/gorilla/websocket"
 	"net/http"
-	"net/url"
-	"strings"
 	"sync"
 )
 
-func initWS() {
+func initWS(origin string) {
 	wsUp = &websocket.Upgrader{
 		ReadBufferSize:  1024,
 		WriteBufferSize: 512000,
 	}
-	wsUp.CheckOrigin = func(r *http.Request) bool {
+
-		origin := r.Header["Origin"]
+	if origin == "*" {
-		if len(origin) == 0 {
+		wsUp.CheckOrigin = func(r *http.Request) bool {
 			return true
 		}
-		o, err := url.Parse(origin[0])
-		if err != nil {
-			return false
-		}
-		if o.Host == r.Host {
-			return true
-		}
-		log.Trace().Msgf("[api.ws] origin: %s, host: %s", o.Host, r.Host)
-		// some users change Nginx external port using Docker port
-		// so origin will be with a port and host without
-		if i := strings.IndexByte(o.Host, ':'); i > 0 {
-			return o.Host[:i] == r.Host
-		}
-		return false
 	}
 }
 

cmd/app/app.go

@@ -10,7 +10,7 @@ import (
 	"runtime"
 )
 
-var Version = "0.1-rc.3"
+var Version = "0.1-rc.4"
 var UserAgent = "go2rtc/" + Version
 
 func Init() {

cmd/ffmpeg/ffmpeg.go

@@ -38,7 +38,7 @@ func Init() {
 		"h264":       "-c:v libx264 -g:v 30 -preset:v superfast -tune:v zerolatency -profile:v main -level:v 4.1 -pix_fmt:v yuv420p",
 		"h264/ultra": "-c:v libx264 -g:v 30 -preset:v ultrafast -tune:v zerolatency",
 		"h264/high":  "-c:v libx264 -g:v 30 -preset:v superfast -tune:v zerolatency",
-		"h265":       "-c:v libx265 -g:v 30 -preset:v ultrafast -tune:v zerolatency",
+		"h265":       "-c:v libx265 -g:v 30 -preset:v superfast -tune:v zerolatency -profile:v main -level:v 5.1 -pix_fmt:v yuv420p",
 		"mjpeg":      "-c:v mjpeg -force_duplicated_matrix:v 1 -huffman:v 0 -pix_fmt:v yuvj420p",
 		"opus":       "-c:a libopus -ar:a 48000 -ac:a 2",
 		"pcmu":       "-c:a pcm_mulaw -ar:a 8000 -ac:a 1",
@@ -146,7 +146,7 @@ func Init() {
 				s += " -vn"
 			case 1:
 				if len(query["audio"]) > 1 {
-					s += " -map 0:v:0"
+					s += " -map 0:v:0?"
 				}
 				for _, video := range query["video"] {
 					if video == "copy" {
@@ -158,9 +158,9 @@ func Init() {
 			default:
 				for i, video := range query["video"] {
 					if video == "copy" {
-						s += " -map 0:v:0 -c:v:" + strconv.Itoa(i) + " copy"
+						s += " -map 0:v:0? -c:v:" + strconv.Itoa(i) + " copy"
 					} else {
-						s += " -map 0:v:0 " + strings.ReplaceAll(tpl[video], ":v ", ":v:"+strconv.Itoa(i)+" ")
+						s += " -map 0:v:0? " + strings.ReplaceAll(tpl[video], ":v ", ":v:"+strconv.Itoa(i)+" ")
 					}
 				}
 			}
@@ -170,7 +170,7 @@ func Init() {
 				s += " -an"
 			case 1:
 				if len(query["video"]) > 1 {
-					s += " -map 0:a:0"
+					s += " -map 0:a:0?"
 				}
 				for _, audio := range query["audio"] {
 					if audio == "copy" {
@@ -182,9 +182,9 @@ func Init() {
 			default:
 				for i, audio := range query["audio"] {
 					if audio == "copy" {
-						s += " -map 0:a:0 -c:a:" + strconv.Itoa(i) + " copy"
+						s += " -map 0:a:0? -c:a:" + strconv.Itoa(i) + " copy"
 					} else {
-						s += " -map 0:a:0 " + strings.ReplaceAll(tpl[audio], ":a ", ":a:"+strconv.Itoa(i)+" ")
+						s += " -map 0:a:0? " + strings.ReplaceAll(tpl[audio], ":a ", ":a:"+strconv.Itoa(i)+" ")
 					}
 				}
 			}

cmd/mp4/mp4.go

@@ -63,12 +63,12 @@ func handlerKeyframe(w http.ResponseWriter, r *http.Request) {
 }
 
 func handlerMP4(w http.ResponseWriter, r *http.Request) {
+	log.Trace().Msgf("[api.mp4] %s %+v", r.Method, r.Header)
+
 	if isChromeFirst(w, r) || isSafari(w, r) {
 		return
 	}
 
-	log.Trace().Msgf("[api.mp4] %+v", r)
-
 	src := r.URL.Query().Get("src")
 	stream := streams.GetOrNew(src)
 	if stream == nil {

cmd/mp4/mse.go

@@ -5,6 +5,7 @@ import (
 	"github.com/AlexxIT/go2rtc/cmd/streams"
 	"github.com/AlexxIT/go2rtc/pkg/mp4"
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
+	"strings"
 )
 
 const MsgTypeMSE = "mse" // fMP4
@@ -22,6 +23,10 @@ func handlerWS(ctx *api.Context, msg *streamer.Message) {
 	cons.UserAgent = ctx.Request.UserAgent()
 	cons.RemoteAddr = ctx.Request.RemoteAddr
 
+	if codecs, ok := msg.Value.(string); ok {
+		cons.Medias = parseMedias(codecs)
+	}
+
 	cons.Listen(func(msg interface{}) {
 		if data, ok := msg.([]byte); ok {
 			for len(data) > packetSize {
@@ -55,3 +60,42 @@ func handlerWS(ctx *api.Context, msg *streamer.Message) {
 
 	cons.Start()
 }
+
+func parseMedias(codecs string) (medias []*streamer.Media) {
+	var videos []*streamer.Codec
+	var audios []*streamer.Codec
+
+	for _, name := range strings.Split(codecs, ",") {
+		switch name {
+		case "avc1.640029":
+			codec := &streamer.Codec{Name: streamer.CodecH264}
+			videos = append(videos, codec)
+		case "hvc1.1.6.L153.B0":
+			codec := &streamer.Codec{Name: streamer.CodecH265}
+			videos = append(videos, codec)
+		case "mp4a.40.2":
+			codec := &streamer.Codec{Name: streamer.CodecAAC}
+			audios = append(audios, codec)
+		}
+	}
+
+	if videos != nil {
+		media := &streamer.Media{
+			Kind:      streamer.KindVideo,
+			Direction: streamer.DirectionRecvonly,
+			Codecs:    videos,
+		}
+		medias = append(medias, media)
+	}
+
+	if audios != nil {
+		media := &streamer.Media{
+			Kind:      streamer.KindAudio,
+			Direction: streamer.DirectionRecvonly,
+			Codecs:    audios,
+		}
+		medias = append(medias, media)
+	}
+
+	return
+}

cmd/rtsp/rtsp.go

@@ -140,6 +140,11 @@ func tcpHandler(c net.Conn) {
 
 		switch msg {
 		case rtsp.MethodDescribe:
+			if len(conn.URL.Path) == 0 {
+				log.Warn().Msg("[rtsp] server empty URL on DESCRIBE")
+				return
+			}
+
 			name = conn.URL.Path[1:]
 
 			stream := streams.Get(name)
@@ -161,6 +166,11 @@ func tcpHandler(c net.Conn) {
 			}
 
 		case rtsp.MethodAnnounce:
+			if len(conn.URL.Path) == 0 {
+				log.Warn().Msg("[rtsp] server empty URL on ANNOUNCE")
+				return
+			}
+
 			name = conn.URL.Path[1:]
 
 			stream := streams.Get(name)

cmd/webrtc/candidates.go

@@ -13,7 +13,27 @@ func AddCandidate(address string) {
 	candidates = append(candidates, address)
 }
 
-func addCanditates(answer string) (string, error) {
+func asyncCandidates(ctx *api.Context) {
+	for _, address := range candidates {
+		address, err := webrtc.LookupIP(address)
+		if err != nil {
+			log.Warn().Err(err).Caller().Send()
+			continue
+		}
+
+		cand, err := webrtc.NewCandidate(address)
+		if err != nil {
+			log.Warn().Err(err).Caller().Send()
+			continue
+		}
+
+		log.Trace().Str("candidate", cand).Msg("[webrtc] config")
+
+		ctx.Write(&streamer.Message{Type: webrtc.MsgTypeCandidate, Value: cand})
+	}
+}
+
+func syncCanditates(answer string) (string, error) {
 	if len(candidates) == 0 {
 		return answer, nil
 	}

cmd/webrtc/webrtc.go

@@ -33,7 +33,7 @@ func Init() {
 	address := cfg.Mod.Listen
 	pionAPI, err := webrtc.NewAPI(address)
 	if pionAPI == nil {
-		log.Error().Err(err).Msg("[webrtc] init API")
+		log.Error().Err(err).Caller().Msg("webrtc.NewAPI")
 		return
 	}
 
@@ -55,7 +55,7 @@ func Init() {
 
 	candidates = cfg.Mod.Candidates
 
-	api.HandleWS(webrtc.MsgTypeOffer, offerHandler)
+	api.HandleWS(webrtc.MsgTypeOffer, asyncHandler)
 	api.HandleWS(webrtc.MsgTypeCandidate, candidateHandler)
 
 	api.HandleFunc("api/webrtc", syncHandler)
@@ -66,7 +66,7 @@ var log zerolog.Logger
 
 var NewPConn func() (*pion.PeerConnection, error)
 
-func offerHandler(ctx *api.Context, msg *streamer.Message) {
+func asyncHandler(ctx *api.Context, msg *streamer.Message) {
 	src := ctx.Request.URL.Query().Get("src")
 	stream := streams.Get(src)
 	if stream == nil {
@@ -81,7 +81,7 @@ func offerHandler(ctx *api.Context, msg *streamer.Message) {
 	conn := new(webrtc.Conn)
 	conn.Conn, err = NewPConn()
 	if err != nil {
-		log.Error().Err(err).Msg("[webrtc] new conn")
+		log.Error().Err(err).Caller().Msg("NewPConn")
 		return
 	}
 
@@ -104,14 +104,14 @@ func offerHandler(ctx *api.Context, msg *streamer.Message) {
 	log.Trace().Msgf("[webrtc] offer:\n%s", offer)
 
 	if err = conn.SetOffer(offer); err != nil {
-		log.Warn().Err(err).Msg("[api.webrtc] set offer")
+		log.Warn().Err(err).Caller().Msg("conn.SetOffer")
 		ctx.Error(err)
 		return
 	}
 
 	// 2. AddConsumer, so we get new tracks
 	if err = stream.AddConsumer(conn); err != nil {
-		log.Warn().Err(err).Msg("[api.webrtc] add consumer")
+		log.Warn().Err(err).Caller().Msg("stream.AddConsumer")
 		_ = conn.Conn.Close()
 		ctx.Error(err)
 		return
@@ -120,25 +120,20 @@ func offerHandler(ctx *api.Context, msg *streamer.Message) {
 	conn.Init()
 
 	// exchange sdp without waiting all candidates
-	//answer, err := conn.ExchangeSDP(offer, false)
+	answer, err := conn.GetAnswer()
-	//answer, err := conn.GetAnswer()
-	answer, err := conn.GetCompleteAnswer()
-	if err == nil {
-		answer, err = addCanditates(answer)
-	}
 	log.Trace().Msgf("[webrtc] answer\n%s", answer)
 
 	if err != nil {
-		log.Error().Err(err).Msg("[webrtc] get answer")
+		log.Error().Err(err).Caller().Msg("conn.GetAnswer")
 		ctx.Error(err)
 		return
 	}
 
-	ctx.Write(&streamer.Message{
-		Type: webrtc.MsgTypeAnswer, Value: answer,
-	})
-
 	ctx.Consumer = conn
+
+	ctx.Write(&streamer.Message{Type: webrtc.MsgTypeAnswer, Value: answer})
+
+	asyncCandidates(ctx)
 }
 
 func syncHandler(w http.ResponseWriter, r *http.Request) {
@@ -151,19 +146,19 @@ func syncHandler(w http.ResponseWriter, r *http.Request) {
 	// get offer
 	offer, err := ioutil.ReadAll(r.Body)
 	if err != nil {
-		log.Error().Err(err).Caller().Send()
+		log.Error().Err(err).Caller().Msg("ioutil.ReadAll")
 		return
 	}
 
 	answer, err := ExchangeSDP(stream, string(offer), r.UserAgent())
 	if err != nil {
-		log.Error().Err(err).Caller().Send()
+		log.Error().Err(err).Caller().Msg("ExchangeSDP")
 		return
 	}
 
 	// send SDP to client
 	if _, err = w.Write([]byte(answer)); err != nil {
-		log.Error().Err(err).Caller().Send()
+		log.Error().Err(err).Caller().Msg("w.Write")
 	}
 }
 
@@ -174,7 +169,7 @@ func ExchangeSDP(
 	conn := new(webrtc.Conn)
 	conn.Conn, err = NewPConn()
 	if err != nil {
-		log.Error().Err(err).Msg("[webrtc] new conn")
+		log.Error().Err(err).Caller().Msg("NewPConn")
 		return
 	}
 
@@ -192,13 +187,13 @@ func ExchangeSDP(
 	log.Trace().Msgf("[webrtc] offer:\n%s", offer)
 
 	if err = conn.SetOffer(offer); err != nil {
-		log.Warn().Err(err).Msg("[api.webrtc] set offer")
+		log.Warn().Err(err).Caller().Msg("conn.SetOffer")
 		return
 	}
 
 	// 2. AddConsumer, so we get new tracks
 	if err = stream.AddConsumer(conn); err != nil {
-		log.Warn().Err(err).Msg("[api.webrtc] add consumer")
+		log.Warn().Err(err).Caller().Msg("stream.AddConsumer")
 		_ = conn.Conn.Close()
 		return
 	}
@@ -209,12 +204,12 @@ func ExchangeSDP(
 	//answer, err := conn.ExchangeSDP(offer, false)
 	answer, err = conn.GetCompleteAnswer()
 	if err == nil {
-		answer, err = addCanditates(answer)
+		answer, err = syncCanditates(answer)
 	}
 	log.Trace().Msgf("[webrtc] answer\n%s", answer)
 
 	if err != nil {
-		log.Error().Err(err).Msg("[webrtc] get answer")
+		log.Error().Err(err).Caller().Msg("conn.GetCompleteAnswer")
 	}
 
 	return

pkg/fake/producer.go

@@ -24,7 +24,7 @@ func (p *Producer) GetTrack(media *streamer.Media, codec *streamer.Codec) *strea
 		panic("you shall not pass!")
 	}
 
-	track := &streamer.Track{Codec: codec, Direction: media.Direction}
+	track := streamer.NewTrack(codec, media.Direction)
 
 	switch media.Direction {
 	case streamer.DirectionSendonly:

pkg/h265/README.md

@@ -1,3 +1,4 @@
 ## Useful links
 
 - https://datatracker.ietf.org/doc/html/rfc7798
+- [Add initial support for WebRTC HEVC](https://trac.webkit.org/changeset/259452/webkit)

pkg/h265/helper.go

@@ -2,19 +2,57 @@ package h265
 
 import (
 	"encoding/base64"
+	"encoding/binary"
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
 )
 
 const (
-	NALUnitTypeIFrame = 19
+	NALUTypePFrame  = 1
+	NALUTypeIFrame  = 19
+	NALUTypeIFrame2 = 20
+	NALUTypeIFrame3 = 21
+	NALUTypeVPS     = 32
+	NALUTypeSPS     = 33
+	NALUTypePPS     = 34
+	NALUTypeFU      = 49
 )
 
-func NALUnitType(b []byte) byte {
+func NALUType(b []byte) byte {
-	return b[4] >> 1
+	return (b[4] >> 1) & 0x3F
 }
 
 func IsKeyframe(b []byte) bool {
-	return NALUnitType(b) == NALUnitTypeIFrame
+	for {
+		switch NALUType(b) {
+		case NALUTypePFrame:
+			return false
+		case NALUTypeIFrame, NALUTypeIFrame2, NALUTypeIFrame3:
+			return true
+		}
+
+		size := int(binary.BigEndian.Uint32(b)) + 4
+		if size < len(b) {
+			b = b[size:]
+			continue
+		} else {
+			return false
+		}
+	}
+}
+
+func Types(data []byte) []byte {
+	var types []byte
+	for {
+		types = append(types, NALUType(data))
+
+		size := 4 + int(binary.BigEndian.Uint32(data))
+		if size < len(data) {
+			data = data[size:]
+		} else {
+			break
+		}
+	}
+	return types
 }
 
 func GetParameterSet(fmtp string) (vps, sps, pps []byte) {

pkg/h265/rtp.go

@@ -1,6 +1,7 @@
 package h265
 
 import (
+	"encoding/binary"
 	"github.com/AlexxIT/go2rtc/pkg/h264"
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
 	"github.com/deepch/vdk/codec/h265parser"
@@ -8,77 +9,59 @@ import (
 )
 
 func RTPDepay(track *streamer.Track) streamer.WrapperFunc {
-	vps, sps, pps := GetParameterSet(track.Codec.FmtpLine)
+	//vps, sps, pps := GetParameterSet(track.Codec.FmtpLine)
+	//ps := h264.EncodeAVC(vps, sps, pps)
 
-	var buffer []byte
+	buf := make([]byte, 0, 512*1024) // 512K
+	var nuStart int
 
 	return func(push streamer.WriterFunc) streamer.WriterFunc {
 		return func(packet *rtp.Packet) error {
-			nut := (packet.Payload[0] >> 1) & 0x3f
+			data := packet.Payload
-			//fmt.Printf(
+			nuType := (data[0] >> 1) & 0x3F
-			//	"[RTP] codec: %s, nalu: %2d, size: %6d, ts: %10d, pt: %2d, ssrc: %d, seq: %d\n",
+			//log.Printf("[RTP] codec: %s, nalu: %2d, size: %6d, ts: %10d, pt: %2d, ssrc: %d, seq: %d, %v", track.Codec.Name, nuType, len(packet.Payload), packet.Timestamp, packet.PayloadType, packet.SSRC, packet.SequenceNumber, packet.Marker)
-			//	track.Codec.Name, nut, len(packet.Payload), packet.Timestamp,
-			//	packet.PayloadType, packet.SSRC, packet.SequenceNumber,
-			//)
 
-			switch nut {
+			if nuType == NALUTypeFU {
-			case h265parser.NAL_UNIT_UNSPECIFIED_49:
-				data := packet.Payload
 				switch data[2] >> 6 {
 				case 2: // begin
-					buffer = []byte{
+					nuType = data[2] & 0x3F
-						(data[0] & 0x81) | (data[2] & 0x3f << 1), data[1],
+
-					}
+					// push PS data before keyframe
-					buffer = append(buffer, data[3:]...)
+					//if len(buf) == 0 && nuType >= 19 && nuType <= 21 {
+					//	buf = append(buf, ps...)
+					//}
+
+					nuStart = len(buf)
+					buf = append(buf, 0, 0, 0, 0) // NAL unit size
+					buf = append(buf, (data[0]&0x81)|(nuType<<1), data[1])
+					buf = append(buf, data[3:]...)
 					return nil
 				case 0: // continue
-					buffer = append(buffer, data[3:]...)
+					buf = append(buf, data[3:]...)
 					return nil
 				case 1: // end
-					packet.Payload = append(buffer, data[3:]...)
+					buf = append(buf, data[3:]...)
+					binary.BigEndian.PutUint32(buf[nuStart:], uint32(len(buf)-nuStart-4))
 				}
-			case h265parser.NAL_UNIT_VPS:
+			} else {
-				vps = packet.Payload
+				nuStart = len(buf)
-				return nil
+				buf = append(buf, 0, 0, 0, 0) // NAL unit size
-			case h265parser.NAL_UNIT_SPS:
+				buf = append(buf, data...)
-				sps = packet.Payload
+				binary.BigEndian.PutUint32(buf[nuStart:], uint32(len(data)))
-				return nil
-			case h265parser.NAL_UNIT_PPS:
-				pps = packet.Payload
-				return nil
-			default:
-				//panic("not implemented")
 			}
 
-			var clone rtp.Packet
+			// collect all NAL Units for Access Unit
-
+			if !packet.Marker {
-			nut = (packet.Payload[0] >> 1) & 0x3f
+				return nil
-			if nut >= h265parser.NAL_UNIT_CODED_SLICE_BLA_W_LP && nut <= h265parser.NAL_UNIT_CODED_SLICE_CRA {
-				clone = *packet
-				clone.Version = h264.RTPPacketVersionAVC
-				clone.Payload = h264.EncodeAVC(vps)
-				if err := push(&clone); err != nil {
-					return err
-				}
-
-				clone = *packet
-				clone.Version = h264.RTPPacketVersionAVC
-				clone.Payload = h264.EncodeAVC(sps)
-				if err := push(&clone); err != nil {
-					return err
-				}
-
-				clone = *packet
-				clone.Version = h264.RTPPacketVersionAVC
-				clone.Payload = h264.EncodeAVC(pps)
-				if err := push(&clone); err != nil {
-					return err
-				}
 			}
 
-			clone = *packet
+			//log.Printf("[HEVC] %v, len: %d", Types(buf), len(buf))
+
+			clone := *packet
 			clone.Version = h264.RTPPacketVersionAVC
-			clone.Payload = h264.EncodeAVC(packet.Payload)
+			clone.Payload = buf
+
+			buf = buf[:0]
 
 			return push(&clone)
 		}
@@ -106,13 +89,13 @@ func SafariPay(mtu uint16) streamer.WrapperFunc {
 
 			var start byte
 
-			nut := (data[4] >> 1) & 0b111111
+			nuType := (data[4] >> 1) & 0b111111
 			//fmt.Printf("[H265] nut: %2d, size: %6d, data: %16x\n", nut, len(data), data[4:20])
 			switch {
-			case nut >= h265parser.NAL_UNIT_VPS && nut <= h265parser.NAL_UNIT_PPS:
+			case nuType >= h265parser.NAL_UNIT_VPS && nuType <= h265parser.NAL_UNIT_PPS:
 				buffer = append(buffer, data...)
 				return nil
-			case nut >= h265parser.NAL_UNIT_CODED_SLICE_BLA_W_LP && nut <= h265parser.NAL_UNIT_CODED_SLICE_CRA:
+			case nuType >= h265parser.NAL_UNIT_CODED_SLICE_BLA_W_LP && nuType <= h265parser.NAL_UNIT_CODED_SLICE_CRA:
 				buffer = append([]byte{3}, buffer...)
 				data = append(buffer, data...)
 				start = 1

pkg/homekit/client.go

@@ -75,7 +75,7 @@ func (c *Client) GetTrack(media *streamer.Media, codec *streamer.Codec) *streame
 		}
 	}
 
-	track := &streamer.Track{Codec: codec, Direction: media.Direction}
+	track := streamer.NewTrack(codec, media.Direction)
 	c.tracks = append(c.tracks, track)
 	return track
 }

pkg/ivideon/client.go

@@ -192,10 +192,7 @@ func (c *Client) getTracks() error {
 				}
 				c.medias = append(c.medias, media)
 
-				track := &streamer.Track{
+				track := streamer.NewTrack(codec, streamer.DirectionSendonly)
-					Direction: streamer.DirectionSendonly,
-					Codec:     codec,
-				}
 				c.tracks[msg.TrackID] = track
 
 			case "mp4a": // mp4a.40.2

pkg/mp4/consumer.go

@@ -12,6 +12,7 @@ import (
 type Consumer struct {
 	streamer.Element
 
+	Medias     []*streamer.Media
 	UserAgent  string
 	RemoteAddr string
 
@@ -23,6 +24,11 @@ type Consumer struct {
 }
 
 func (c *Consumer) GetMedias() []*streamer.Media {
+	if c.Medias != nil {
+		return c.Medias
+	}
+
+	// default medias
 	return []*streamer.Media{
 		{
 			Kind:      streamer.KindVideo,

pkg/mp4/muxer.go

@@ -3,7 +3,6 @@ package mp4
 import (
 	"encoding/binary"
 	"encoding/hex"
-	"fmt"
 	"github.com/AlexxIT/go2rtc/pkg/h264"
 	"github.com/AlexxIT/go2rtc/pkg/h265"
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
@@ -36,8 +35,9 @@ func (m *Muxer) MimeType(codecs []*streamer.Codec) string {
 		case streamer.CodecH264:
 			s += "avc1." + h264.GetProfileLevelID(codec.FmtpLine)
 		case streamer.CodecH265:
-			// +Safari +Chrome +Edge -iOS15 -Android13
+			// H.265 profile=main level=5.1
-			s += "hvc1.1.6.L93.B0" // hev1.1.6.L93.B0
+			// hvc1 - supported in Safari, hev1 - doesn't, both supported in Chrome
+			s += "hvc1.1.6.L153.B0"
 		case streamer.CodecAAC:
 			s += "mp4a.40.2"
 		}
@@ -97,7 +97,10 @@ func (m *Muxer) GetInit(codecs []*streamer.Codec) ([]byte, error) {
 		case streamer.CodecH265:
 			vps, sps, pps := h265.GetParameterSet(codec.FmtpLine)
 			if sps == nil {
-				return nil, fmt.Errorf("empty SPS: %#v", codec)
+				// some dummy SPS and PPS not a problem
+				vps = []byte{0x40, 0x01, 0x0c, 0x01, 0xff, 0xff, 0x01, 0x40, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x99, 0xac, 0x09}
+				sps = []byte{0x42, 0x01, 0x01, 0x01, 0x40, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x99, 0xa0, 0x01, 0x40, 0x20, 0x05, 0xa1, 0xfe, 0x5a, 0xee, 0x46, 0xc1, 0xae, 0x55, 0x04}
+				pps = []byte{0x44, 0x01, 0xc0, 0x73, 0xc0, 0x4c, 0x90}
 			}
 
 			codecData, err := h265parser.NewCodecDataFromVPSAndSPSAndPPS(vps, sps, pps)

pkg/rtmp/client.go

@@ -83,9 +83,7 @@ func (c *Client) Dial() (err error) {
 			}
 			c.medias = append(c.medias, media)
 
-			track := &streamer.Track{
+			track := streamer.NewTrack(codec, media.Direction)
-				Codec: codec, Direction: media.Direction,
-			}
 			c.tracks = append(c.tracks, track)
 
 		case av.AAC:
@@ -108,9 +106,7 @@ func (c *Client) Dial() (err error) {
 			}
 			c.medias = append(c.medias, media)
 
-			track := &streamer.Track{
+			track := streamer.NewTrack(codec, media.Direction)
-				Codec: codec, Direction: media.Direction,
-			}
 			c.tracks = append(c.tracks, track)
 
 		default:

pkg/rtsp/conn.go

@@ -429,9 +429,7 @@ func (c *Conn) SetupMedia(
 		return nil, err
 	}
 
-	track := &streamer.Track{
+	track := streamer.NewTrack(codec, media.Direction)
-		Codec: codec, Direction: media.Direction,
-	}
 
 	switch track.Direction {
 	case streamer.DirectionSendonly:
@@ -519,9 +517,7 @@ func (c *Conn) Accept() error {
 			// TODO: fix someday...
 			c.channels = map[byte]*streamer.Track{}
 			for i, media := range c.Medias {
-				track := &streamer.Track{
+				track := streamer.NewTrack(media.Codecs[0], media.Direction)
-					Codec: media.Codecs[0], Direction: media.Direction,
-				}
 				c.tracks = append(c.tracks, track)
 				c.channels[byte(i<<1)] = track
 			}

pkg/streamer/track.go

@@ -13,12 +13,18 @@ type Track struct {
 	Codec     *Codec
 	Direction string
 	sink      map[*Track]WriterFunc
-	sinkMu    sync.RWMutex
+	sinkMu    *sync.RWMutex
+}
+
+func NewTrack(codec *Codec, direction string) *Track {
+	return &Track{Codec: codec, Direction: direction, sinkMu: new(sync.RWMutex)}
 }
 
 func (t *Track) String() string {
 	s := t.Codec.String()
+	t.sinkMu.RLock()
 	s += fmt.Sprintf(", sinks=%d", len(t.sink))
+	t.sinkMu.RUnlock()
 	return s
 }
 
@@ -38,14 +44,12 @@ func (t *Track) Bind(w WriterFunc) *Track {
 		t.sink = map[*Track]WriterFunc{}
 	}
 
-	clone := &Track{
+	clone := *t
-		Codec: t.Codec, Direction: t.Direction, sink: t.sink,
+	t.sink[&clone] = w
-	}
-	t.sink[clone] = w
 
 	t.sinkMu.Unlock()
 
-	return clone
+	return &clone
 }
 
 func (t *Track) Unbind() {
@@ -55,7 +59,9 @@ func (t *Track) Unbind() {
 }
 
 func (t *Track) GetSink(from *Track) {
+	t.sinkMu.Lock()
 	t.sink = from.sink
+	t.sinkMu.Unlock()
 }
 
 func (t *Track) HasSink() bool {

pkg/webrtc/conn.go

@@ -1,10 +1,8 @@
 package webrtc
 
 import (
-	"fmt"
 	"github.com/AlexxIT/go2rtc/pkg/streamer"
 	"github.com/pion/webrtc/v3"
-	"sort"
 )
 
 const (
@@ -59,7 +57,7 @@ func (c *Conn) Init() {
 			}
 		}
 
-		fmt.Printf("TODO: webrtc ontrack %+v\n", remote)
+		//fmt.Printf("TODO: webrtc ontrack %+v\n", remote)
 	})
 
 	// OK connection:
@@ -100,12 +98,23 @@ func (c *Conn) SetOffer(offer string) (err error) {
 		return
 	}
 	rawSDP := []byte(c.Conn.RemoteDescription().SDP)
-	c.medias, err = streamer.UnmarshalSDP(rawSDP)
+	medias, err := streamer.UnmarshalSDP(rawSDP)
+	if err != nil {
+		return
+	}
 
 	// sort medias, so video will always be before audio
-	sort.Slice(c.medias, func(i, j int) bool {
+	// and ignore application media from Hass default lovelace card
-		return c.medias[i].Kind == streamer.KindVideo
+	for _, media := range medias {
-	})
+		if media.Kind == streamer.KindVideo {
+			c.medias = append(c.medias, media)
+		}
+	}
+	for _, media := range medias {
+		if media.Kind == streamer.KindAudio {
+			c.medias = append(c.medias, media)
+		}
+	}
 
 	return
 }

www/mse.html

@@ -3,97 +3,26 @@
 <head>
     <meta name="viewport" content="width=device-width, initial-scale=1">
     <title>go2rtc - MSE</title>
+    <script src="video-rtc.js"></script>
     <style>
         body {
+            background: black;
             margin: 0;
             padding: 0;
         }
 
-        html, body {
+        html, body, video {
             height: 100%;
             width: 100%;
         }
-
-        #video {
-            width: 100%;
-            height: 100%;
-            background: black;
-        }
     </style>
 </head>
 <body>
-<!-- muted is important for autoplay -->
-<video id="video" autoplay controls playsinline muted></video>
 <script>
-    // support api_path
+    const url = new URL("api/ws" + location.search, location.href);
-    const baseUrl = location.origin + location.pathname.substr(
+    const video = document.createElement("video-rtc");
-        0, location.pathname.lastIndexOf("/")
+    video.src = url.toString();
-    );
+    document.body.appendChild(video);
-    const video = document.querySelector('#video');
-
-    function init() {
-        let mediaSource, sourceBuffer, queueBuffer = [];
-
-        const ws = new WebSocket(`ws${baseUrl.substr(4)}/api/ws${location.search}`);
-        ws.binaryType = "arraybuffer";
-
-        ws.onopen = () => {
-            mediaSource = new MediaSource();
-            video.src = URL.createObjectURL(mediaSource);
-            mediaSource.onsourceopen = () => {
-                mediaSource.onsourceopen = null;
-                URL.revokeObjectURL(video.src);
-                ws.send(JSON.stringify({"type": "mse"}));
-            };
-        };
-
-        ws.onmessage = ev => {
-            if (typeof ev.data === 'string') {
-                const data = JSON.parse(ev.data);
-                console.debug("ws.onmessage", data);
-
-                if (data.type === "mse") {
-                    sourceBuffer = mediaSource.addSourceBuffer(data.value);
-                    sourceBuffer.mode = "segments"; // segments or sequence
-                    sourceBuffer.onupdateend = () => {
-                        if (!sourceBuffer.updating && queueBuffer.length > 0) {
-                            try {
-                                sourceBuffer.appendBuffer(queueBuffer.shift());
-                            } catch (e) {
-                                // console.warn(e);
-                            }
-                        }
-                    }
-                }
-            } else if (sourceBuffer.updating || queueBuffer.length > 0) {
-                queueBuffer.push(ev.data);
-            } else {
-                try {
-                    sourceBuffer.appendBuffer(ev.data);
-                } catch (e) {
-                    // console.warn(e);
-                }
-            }
-
-            if (video.seekable.length > 0) {
-                const delay = video.seekable.end(video.seekable.length - 1) - video.currentTime;
-                if (delay < 1) {
-                    video.playbackRate = 1;
-                } else if (delay > 10) {
-                    video.playbackRate = 10;
-                } else if (delay > 2) {
-                    video.playbackRate = Math.floor(delay);
-                }
-            }
-        }
-
-        video.onpause = () => {
-            ws.close();
-            setTimeout(init, 0);
-        }
-    }
-
-    init();
 </script>
 </body>
 </html>

www/video-rtc.js

@@ -0,0 +1,374 @@
+/**
+ * Common function for processing MSE and MSE2 data.
+ * @param ms MediaSource
+ */
+function MediaSourceHandler(ms) {
+    let sb, qb = [];
+
+    return ev => {
+        if (typeof ev.data === "string") {
+            const msg = JSON.parse(ev.data);
+            if (msg.type === "mse") {
+                if (!MediaSource.isTypeSupported(msg.value)) {
+                    console.warn("Not supported: " + msg.value)
+                    return;
+                }
+
+                sb = ms.addSourceBuffer(msg.value);
+                sb.mode = "segments"; // segments or sequence
+                sb.addEventListener("updateend", () => {
+                    if (!sb.updating && qb.length > 0) {
+                        try {
+                            sb.appendBuffer(qb.shift());
+                        } catch (e) {
+                            // console.warn(e);
+                        }
+                    }
+                });
+            }
+        } else if (sb.updating || qb.length > 0) {
+            qb.push(ev.data);
+            // console.debug("buffer:", qb.length);
+        } else {
+            try {
+                sb.appendBuffer(ev.data);
+            } catch (e) {
+                // console.warn(e);
+            }
+        }
+    }
+}
+
+/**
+ * Dedicated Worker Handler for MSE2 https://chromestatus.com/feature/5177263249162240
+ */
+if (typeof importScripts == "function") {
+    // protect below code (class VideoRTC) from fail inside Worker
+    HTMLElement = Object;
+    customElements = {define: Function()};
+
+    const ms = new MediaSource();
+    ms.addEventListener("sourceopen", ev => {
+        postMessage({type: ev.type});
+    }, {once: true});
+
+    onmessage = MediaSourceHandler(ms);
+
+    postMessage({type: "handle", value: ms.handle}, [ms.handle]);
+}
+
+/**
+ * Video player for MSE and WebRTC connections.
+ *
+ * All modern web technologies are supported in almost any browser except Apple Safari.
+ *
+ * Support:
+ * - RTCPeerConnection for Safari iOS 11.0+
+ * - IntersectionObserver for Safari iOS 12.2+
+ * - MediaSource in Workers for Chrome 108+
+ *
+ * Doesn't support:
+ * - MediaSource for Safari iOS all
+ * - Customized built-in elements (extends HTMLVideoElement) because all Safari
+ */
+class VideoRTC extends HTMLElement {
+    DISCONNECT_TIMEOUT = 5000;
+    RECONNECT_TIMEOUT = 30000;
+
+    CODECS = [
+        "avc1.640029",      // H.264 high 4.1 (Chromecast 1st and 2nd Gen)
+        "avc1.64002A",      // H.264 high 4.2 (Chromecast 3rd Gen)
+        "hvc1.1.6.L153.B0", // H.265 main 5.1 (Chromecast Ultra)
+        "mp4a.40.2",        // AAC LC
+        "mp4a.40.5",        // AAC HE
+        "mp4a.69",          // MP3
+        "mp4a.6B",          // MP3
+    ];
+
+    /**
+     * Enable MediaSource in Workers mode.
+     * @type {boolean}
+     */
+    MSE2 = true;
+
+    /**
+     * Run stream when not displayed on the screen. Default `false`.
+     * @type {boolean}
+     */
+    background = false;
+
+    /**
+     * Run stream only when player in the viewport. Stop when user scroll out player.
+     * Value is percentage of visibility from `0` (not visible) to `1` (full visible).
+     * Default `0` - disable;
+     * @type {number}
+     */
+    intersectionThreshold = 0;
+
+    /**
+     * Run stream only when browser page on the screen. Stop when user change browser
+     * tab or minimise browser windows.
+     * @type {boolean}
+     */
+    visibilityCheck = true;
+
+    /**
+     * @type {HTMLVideoElement}
+     */
+    video = null;
+
+    /**
+     * @type {RTCPeerConnection}
+     */
+    pc = null;
+
+    /**
+     * @type {WebSocket}
+     */
+    ws = null;
+
+    /**
+     * Internal WebSocket connection state. Values: CLOSED, CONNECTING, OPEN
+     * @type {number}
+     */
+    wsState = WebSocket.CLOSED;
+
+    /**
+     * Internal WebSocket URL.
+     * @type {string}
+     */
+    wsURL = "";
+
+    /**
+     * Internal disconnect TimeoutID.
+     * @type {number}
+     */
+    disconnectTimeout = 0;
+
+    /**
+     * Internal reconnect TimeoutID.
+     * @type {number}
+     */
+    reconnectTimeout = 0;
+
+    constructor() {
+        super();
+
+        console.debug("this.constructor");
+
+        this.video = document.createElement("video");
+        this.video.controls = true;
+        this.video.playsInline = true;
+    }
+
+    /** public properties **/
+
+    /**
+     * Set video source (WebSocket URL). Support relative path.
+     * @param value
+     */
+    set src(value) {
+        if (value.startsWith("/")) {
+            value = "ws" + location.origin.substr(4) + value;
+        } else if (value.startsWith("http")) {
+            value = "ws" + value.substr(4);
+        }
+
+        this.wsURL = value;
+
+        if (this.isConnected) this.connectedCallback();
+    }
+
+    /**
+     * Play video. Support automute when autoplay blocked.
+     * https://developer.chrome.com/blog/autoplay/
+     */
+    play() {
+        this.video.play().catch(er => {
+            if (er.name === "NotAllowedError" && !this.video.muted) {
+                this.video.muted = true;
+                this.video.play();
+            }
+        });
+    }
+
+    get codecs() {
+        return this.CODECS.filter(value => {
+            return MediaSource.isTypeSupported(`video/mp4; codecs="${value}"`);
+        }).join();
+    }
+
+    /**
+     * `CustomElement`. Invoked each time the custom element is appended into a
+     * document-connected element.
+     */
+    connectedCallback() {
+        console.debug("this.connectedCallback", this.wsState);
+        if (this.disconnectTimeout) {
+            clearTimeout(this.disconnectTimeout);
+            this.disconnectTimeout = 0;
+        }
+
+        // because video autopause on disconnected from DOM
+        const seek = this.video.seekable;
+        if (seek.length > 0) {
+            this.video.currentTime = seek.end(seek.length - 1);
+            this.play();
+        }
+
+        if (!this.wsURL || this.wsState !== WebSocket.CLOSED) return;
+
+        // CLOSED => CONNECTING
+        this.wsState = WebSocket.CONNECTING;
+
+        this.internalInit();
+        this.internalConnect();
+    }
+
+    /**
+     * `CustomElement`. Invoked each time the custom element is disconnected from the
+     * document's DOM.
+     */
+    disconnectedCallback() {
+        console.debug("this.disconnectedCallback", this.wsState);
+        if (this.background || this.disconnectTimeout ||
+            this.wsState === WebSocket.CLOSED) return;
+
+        this.disconnectTimeout = setTimeout(() => {
+            if (this.reconnectTimeout) {
+                clearTimeout(this.reconnectTimeout);
+                this.reconnectTimeout = 0;
+            }
+
+            this.disconnectTimeout = 0;
+            // CONNECTING, OPEN => CLOSED
+            this.wsState = WebSocket.CLOSED;
+
+            if (this.ws) {
+                this.ws.close();
+                this.ws = null;
+            }
+        }, this.DISCONNECT_TIMEOUT);
+    }
+
+    internalInit() {
+        if (this.childElementCount) return;
+
+        this.appendChild(this.video);
+
+        if (this.background) return;
+
+        if ("hidden" in document && this.visibilityCheck) {
+            document.addEventListener("visibilitychange", () => {
+                if (document.hidden) {
+                    this.disconnectedCallback();
+                } else if (this.isConnected) {
+                    this.connectedCallback();
+                }
+            })
+        }
+
+        if ("IntersectionObserver" in window && this.intersectionThreshold) {
+            const observer = new IntersectionObserver(entries => {
+                entries.forEach(entry => {
+                    if (!entry.isIntersecting) {
+                        this.disconnectedCallback();
+                    } else if (this.isConnected) {
+                        this.connectedCallback();
+                    }
+                });
+            }, {threshold: this.intersectionThreshold});
+            observer.observe(this);
+        }
+    }
+
+    internalConnect() {
+        if (this.wsState !== WebSocket.CONNECTING) return;
+        if (this.ws) throw "connect with non null WebSocket";
+
+        const ts = Date.now();
+
+        this.ws = new WebSocket(this.wsURL);
+        this.ws.binaryType = "arraybuffer";
+
+        this.ws.addEventListener("open", () => {
+            console.debug("ws.open", this.wsState);
+            if (this.wsState !== WebSocket.CONNECTING) return;
+
+            // CONNECTING => OPEN
+            this.wsState = WebSocket.OPEN;
+        });
+        this.ws.addEventListener("close", () => {
+            console.debug("ws.close", this.wsState);
+            if (this.wsState === WebSocket.CLOSED) return;
+
+            // CONNECTING, OPEN => CONNECTING
+            this.wsState = WebSocket.CONNECTING;
+            this.ws = null;
+
+            // reconnect no more than once every X seconds
+            const delay = Math.max(this.RECONNECT_TIMEOUT - (Date.now() - ts), 0);
+
+            this.reconnectTimeout = setTimeout(() => {
+                this.reconnectTimeout = 0;
+                this.internalConnect();
+            }, delay);
+        });
+
+        if ("MediaSource" in window && this.MSE2) {
+            if (MediaSource.canConstructInDedicatedWorker) {
+                this.internalMSE2();
+            } else {
+                this.internalMSE();
+            }
+        }
+
+        // TODO: this.internalRTC();
+    }
+
+    internalMSE() {
+        console.debug("this.internalMSE");
+        this.ws.addEventListener("open", () => {
+            const ms = new MediaSource();
+            ms.addEventListener("sourceopen", () => {
+                URL.revokeObjectURL(this.video.src);
+                this.ws.send(JSON.stringify({type: "mse", value: this.codecs}));
+            }, {once: true});
+
+            this.video.src = URL.createObjectURL(ms);
+            this.play();
+
+            this.ws.addEventListener("message", MediaSourceHandler(ms));
+        });
+    }
+
+    internalMSE2() {
+        console.debug("this.internalMSE2");
+        const worker = new Worker("video-rtc.js");
+        worker.addEventListener("message", ev => {
+            if (ev.data.type === "handle") {
+                this.video.srcObject = ev.data.value;
+                this.play();
+            } else if (ev.data.type === "sourceopen") {
+                this.ws.send(JSON.stringify({type: "mse", value: this.codecs}));
+            }
+        });
+
+        this.ws.addEventListener("message", ev => {
+            if (typeof ev.data === "string") {
+                worker.postMessage(ev.data);
+            } else {
+                worker.postMessage(ev.data, [ev.data]);
+            }
+        });
+        this.ws.addEventListener("close", () => {
+            worker.terminate();
+        });
+    }
+
+    internalRTC() {
+        if (!("RTCPeerConnection" in window)) return; // macOS Desktop app
+    }
+}
+
+customElements.define("video-rtc", VideoRTC);