[Misc]: Missing charset handling

[Cyrille37]

Describe the bug

Hi.
Thanks a lot for sharing this useful and great work. 💌

version: JohannesKaufmann/html-to-markdown/v2 v2.5.0

The library seems to not manage charset declared in html page. If a page declares "" the resulted .md is "iso-8859-1" encoded but we can't know that. Perhaps the output should be "utf-8" encoded. It needs the library manage the page declared charset.

Regards & cheers.

HTML Input

Libérez votre créativité

Generated Markdown

Lib�rez votre cr�ativit�

Expected Markdown

Libérez votre créativité

What plugins did you use?

base, commonmark

[Cyrille37]

Oups! Perhaps not a bug because it is the library user to manage the html encoding before calling conv.ConvertNode().

I solved the problem with :

	htmlBytes, err := os.ReadFile(*filePath)
	if err != nil {
		log.Fatalf("Failed to read file %s: %v", *filePath, err)
	}

	// Parse HTML into DOM
	var doc *html.Node
	if utf8.Valid(htmlBytes) {
		// Already valid UTF-8 — no charset decoding needed
		doc, err = html.Parse(bytes.NewReader(htmlBytes))
	} else {
		// Not valid UTF-8 — detect charset from meta tags and decode
		utf8Reader, decErr := charset.NewReader(bytes.NewReader(htmlBytes), "")
		if decErr != nil {
			log.Fatalf("Failed to detect charset: %v", decErr)
		}
		doc, err = html.Parse(utf8Reader)
	}
	if err != nil {
		log.Fatalf("Failed to parse HTML: %v", err)
	}

This code as to read all file because charset.DetermineEncoding (from golang.org/x/net) only looks at the first 1024 bytes and some html page have long header with a declared charset far away :-(

It would be nice to add a brief note on this in the README ;-)

Cheers.