Discussing Merlin Team's experience report

Challenges faced, design interations and interesting tricks

Hi, I'm Manas.

prometheansacrifice

Reason and OCaml devtools

Maintaining esy

Worked with Ligo team

I helped them build a package manager and ship multi-platform builds

Briefly worked on Tezos core

Again devtools and a core economic protocol proposal

Currently helping setup an OCaml devteam

And hopefully develop an open source template for other companies to follow.

Also run Reason OCaml India virtual meetups

Evolution of Merlin interaction models Model

Adapting OCaml frontend for Merlin

Lexing

Traditional lexers

(* .mll file *)
{
  open Parser        (* The type token is defined in parser.mli *)
}
rule read = parse
[' ' '\t']   { read lexbuf }     (* skip blanks *)
| ['(']      { LPAREN }
| [')']      { RPAREN }

(* Consuming files *)
let lexbuf = Lexing.from_string str in
Parser.main Lexer.read lexbuf

But editors need to collect all the errors

let lexer =
  let keywords = Extension.keywords Mconfig.(config.merlin.extensions) in
  Mreader_lexer.make Mconfig.(config.ocaml.warnings) keywords config source
in
let lexer_keywords = Mreader_lexer.keywords lexer
and lexer_errors = Mreader_lexer.errors lexer
and comments = Mreader_lexer.comments lexer

How does Merlin achieve this?

let get_tokens keywords pos text =
  let state = Lexer_raw.make keywords in
  let lexbuf = Lexing.from_string text in
  Lexing.move lexbuf pos;
  let rec aux items = function
    | Lexer_raw.Return (Parser_raw.COMMENT comment) ->
      continue (Comment comment :: items)
    | Lexer_raw.Refill k -> aux items (k ())
    | Lexer_raw.Return t ->
      let triple = (t, lexbuf.Lexing.lex_start_p, lexbuf.Lexing.lex_curr_p) in
      let items = Triple triple :: items in
      if t = Parser_raw.EOF
      then items
      else continue items
    | Lexer_raw.Fail (err, loc) ->
      continue (Error (err, loc) :: items)

Parsing

Traditional Parsers

Parser.main Lexer.read lexbuf

Desirable Parser API for editors

let lexer =
  let keywords = Extension.keywords Mconfig.(config.merlin.extensions) in
  Mreader_lexer.make Mconfig.(config.ocaml.warnings) keywords config source
in
let parser = Mreader_parser.make Mconfig.(config.ocaml.warnings) lexer kind in
and parser_errors = Mreader_parser.errors parser
and parsetree = Mreader_parser.result parser

Typing

Merlin features

type-at-cursor

Auto Completions

go-to-definition

Destructuring

Polarity search

Challenges

Merlin at scale: the short-paths example

Supporting secondary frontends: Reason

Editor Integrations: poor APIs

Thoughts on Merlin Engineering

No Spec. No tests.

Sensitive to user expections

…more than the compiler or other batch tools.

Too many build systems

Lacks uniform solution to create .merlin files

Future work

Menhir SDK

Cached LR reductions

Thank you

I'm @ManasJayanth on x.com.

Email: manas [at] dining-philosophers [dot] in

These slides can be found at slides.manas-jayanth.com