Yesterday on Reddit, someone asked the following question:

What exactly is the point of recursion schemes?

[..]

I understand it’s usefulness from an academic perspective, understanding the similarities between different types of recursive algorithms and their common patterns, but in practice, it just seems kind of… pointless?

[..]

I’m not not ripping on this particular library or it’s author at all, I think it’s worthwhile to explore topics like this, but as anything other than a mental exercise I struggle to see what the benefit is in practice. Personally I don’t understand what the aversion is to explicit recursion when the algorithm you’re writing calls for it. For other complicated ideas born out of Haskell like lenses I can immediately see the utility that outweighs any added complexity. Am I missing something here?

I thought the same before working on variants and in particular on EADTs. This post is my answer to the question for the posterity and so that I can easily find it back. It’s a revision of my answer on Reddit.

I’ve found a particularly compelling use case for them in my variant package. I’ll try to explain but it’s a bit long to setup…

The idea of variant is to compose a sum type from other types like this:

type MyMaybe a = V [(), a]
type MyEither a b = V [a,b]

At some point we may want to use variants to encode ASTs that can get or lose constructors. E.g.

data Let b e = Let ...
data Abs b e = Abs ...
data App   e = App ...
data Var b   = Var ...

type LC        b e = V [Abs b e, App e, Var b]
type LCWithLet b e = V [Abs b e, App e, Var b, Let b e]

However e in LC b e/LCWithLet b e should really be LC b e/LCWithLet b e (respectively): e is the type of an expression in our AST. But we can’t encode a recursive type like this with simple type synonyms: we need to introduce a fixpoint data type.

newtype Fix f = Fix (f (Fix f))

type LCF        b e = V [Abs b e, App e, Var b]
type LCWithLetF b e = V [Abs b e, App e, Var b, Let b e]

-- our recursive variant-based ASTs
type LC        b = Fix (LCF b)
type LCWithLet b = Fix (LCWithLetF b)

Now to avoid this boilerplate, in my package there is:

-- same as (V xs) but takes an additional argument
-- 'e' applied to every types in the list
newtype VariantF (xs :: [t -> Type]) (e :: t)
   = VariantF (V (ApplyAll e xs))

-- fixpoint datatype specialized for variants
newtype EADT fs
   = EADT (VariantF fs (EADT fs))

It means that we can define a recursive ADT based on variants (i.e. an EADT, for Extensible ADT) like this:

-- every type embedded in an EADT requires the `e`
-- argument (ADT type), even if it doesn't use it.
-- So rewrite "data Var b = Var ..." into:
data Var b e = Var ...

type LC        b = EADT [Abs b, App, Var b]
type LCWithLet b = EADT [Abs b, App, Var b, Let b]

Now suppose we want to write some code to collect the free variables. We can use explicit recursion and write two functions for the two ASTs. But it would be duplicating a lot of code (especially if we have even more ASTs). This is (finally!) where recursion schemes are useful!

It’s easy to add functor instances to the datatypes embedded in the EADTs: just derive them:

data Let b e = Let ... deriving (Functor)
data Abs b e = Abs ... deriving (Functor)
data App   e = App ... deriving (Functor)
data Var b e = Var ... deriving (Functor)

VariantF also has a Functor instance. It’s Functors all the way down!

So we can define a generic function for all those functors using a typeclass like this:

class FreeVars b f where
  -- return free variables. Use this with a bottom-up traversal (catamorphism)
  freeVarsF :: f (Set b) -> Set b

-- see below for the instances

And then we finally use the catamorphism recursion-scheme (renamed to bottomUp) to write our generic freeVars function:

freeVars :: forall b xs. BottomUpF (FreeVars b) xs => EADT xs -> Set b
freeVars xs = bottomUp (toBottomUp @(FreeVars b) freeVarsF) xs
  -- "bottomUp" is just "cata" with a less cryptic name 

A fully working example:

#!/usr/bin/env cabal
{- cabal:
build-depends: base, containers, variant
-}  
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE DeriveFunctor #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE TemplateHaskell #-}
{-# LANGUAGE KindSignatures #-}
{-# LANGUAGE PatternSynonyms #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE TypeApplications #-}
{-# LANGUAGE ScopedTypeVariables #-}

module Main where

import Data.Variant.EADT
import Data.Variant.EADT.TH
import Data.Set (Set)
import qualified Data.Set as Set

-- Definitions of the EADTs

data VarF b e = VarF b     deriving (Functor)
data AbsF b e = AbsF b e   deriving (Functor)
data AppF   e = AppF e e   deriving (Functor)
data LetF b e = LetF b e e deriving (Functor)

eadtPattern 'VarF "Var"
eadtPattern 'AbsF "Abs"
eadtPattern 'AppF "App"
eadtPattern 'LetF "Let"

type LC        b = EADT [VarF b, AbsF b, AppF]
type LCWithLet b = EADT [VarF b, AbsF b, AppF, LetF b]


-- Generic traversal for FreeVars

class FreeVars b f where
  -- | Compute free variables set
  freeVarsF :: f (Set b) -> Set b

instance FreeVars b (VarF b) where
  freeVarsF (VarF b) = Set.singleton b

instance Eq b => FreeVars b (AbsF b) where
  freeVarsF (AbsF b vs) = Set.filter (/= b) vs

instance (Eq b, Ord b) => FreeVars b AppF where
  freeVarsF (AppF vs1 vs2) = Set.union vs1 vs2

instance (Eq b, Ord b) => FreeVars b (LetF b) where
  freeVarsF (LetF b vs1 vs2) = Set.filter (/= b) (Set.union vs1 vs2)

freeVars :: forall b xs. BottomUpF (FreeVars b) xs => EADT xs -> Set b
freeVars xs = bottomUp (toBottomUp @(FreeVars b) freeVarsF) xs
  -- "bottomUp" is just "cata"


-- Examples

type Name = String

example1 :: LC Name
example1 = Abs "x" (Var "k" `App` (Var "(+)" `App` Var "foo" `App` Var "x"))

example2 :: LCWithLet Name
example2 = Let "foo" (Var "(+)" `App` Var "a" `App` Var "b") $
             Abs "x" (Var "k" `App` (Var "(+)" `App` Var "foo" `App` Var "x")) 

example3 :: LCWithLet Name
example3 = Let "foo" (Var "(+)" `App` Var "a" `App` Var "b") $
            liftEADT example1

main :: IO ()
main = do
  -- call `freeVars` on different AST types
  print (freeVars @Name example1)
  print (freeVars @Name example2)
  print (freeVars @Name example3)

When executed it prints this expected result:

> ./Main.hs
fromList ["(+)","foo","k"]
fromList ["(+)","a","b","k"]
fromList ["(+)","a","b","k"]

Changelog

• 2024-08-08: added mention of cabal#2965, real world issue with plugins, and Teofil’s answer.

There are lots of discussions about GHC’s “stability”, making base “reinstallable”, and similar topics. As we’ve been harping on about these topics for years now, I figured I would try to summarize the issues and the current status here.

What are the issues?

A compiler LC for a language L takes some source code SC written in L (hopefully) and generates object code OC for a target platform T. The compiler runs on an host platform H.

We’ll use and abuse of the following notation to summarize this:

SC(L) --LC(H)--> OC(T)

The issue is that GHC is different from this in two aspects:

1. L isn’t independent of LC
2. OC(T) must be linkable with LC(H) (by default)
   • This implies some constraints on L too, so double on (1)

Let’s discuss this more.

L isn’t independent of LC

A new version of GHC supports only a “new” Haskell language. There may be backward incompatible changes in the compiler (e.g. to the way GHC typechecks programs), but these are rare. What’s more frequent is that the standard API offered to Haskell programs (i.e. boot libraries such as base) changes with the compiler.

As a result, a program compiling with GHC version N may not compile with GHC version N+1. It’s as if the language was versionned, as illustrated below.

We have some source code SC compiling with LC-1.0:

SC(L-1.0) --LC-1.0(H)--> OC(T)

A new compiler is released. Let's try to use it!

SC(L-1.0) --LC-2.0(H)--> FAILURE

We need to tweak the source code to make it compilable with LC-2.0:

SC-2.0(L-2.0) --LC-2.0(H)--> OC(T)

The issue is: we can’t use a new compiler without modifying the source code we want to compile. As a result:

• we can’t know if regressions arise from code changes or compiler upgrade.
• as changing code can be costly (all the dependencies need to be fixed in cascade…), some code bases are stuck with old compiler versions and new compilers get less testing than we would hope for.

The project to fix this is referred to as reinstallable base. The idea is to decouple the base library from the GHC compiler used:

• Let’s call ghc-base the standard library provided by GHC and versioned with it.
  • Note: currently ghc-base is split into ghc-prim, ghc-bignum, and ghc-internal.

• ghc-base changes with every GHC release to add/remove/rename/deprecate/modify stuff

• base abstracts over changes in ghc-base for some releases, with some overlap to allow incremental updates: e.g.

base-1.0 depends on ghc-base >= 8.10 && < 9.8
base-2.0 depends on ghc-base >= 9.6 && < 9.10
base-3.0 depends on ghc-base >= 9.8 && < 9.12

When a new ghc-base is released, existing base releases may be adapted to support it or not:

base-1.0 depends on ghc-base >= 8.10 && < 9.8
base-2.0 depends on ghc-base >= 9.6 && < 9.10
base-3.0.1 depends on ghc-base >= 9.8 && < 9.14

It’s up to base maintainers to decide how many releases they want to support and when to make major breaking releases. But:

• base support for ghc-base should overlap to allow bumping base without bumping ghc-base, and vice-versa.
• as GHC depends on base, there should always be a base version supporting the development version of ghc-base so that it can bootstrap itself (maybe with --allow-newer set only for ghc-base when compiling base for GHC).

Now if we have some source code depending on base (and not ghc-base):

• we can switch to a new compiler (i.e. bumping ghc-base too) without modifying the code, as long as the base version used supports the ghc-base version provided by the new compiler.

• we can switch to a new version of base without switching to a new compiler, as long as the new base version used support the ghc-base version provided by the current compiler.

Template Haskell

ghc-base contains the data types expected by the compiler for the abstract syntax of Template Haskell.

So, similarly to base, the template-haskell package also needs to be versionned to support several ghc-base versions so that packages using template-haskell aren’t tied to a specific ghc-base version through their template-haskell dependency:

template-haskell-1.0 depends on ghc-base >= 8.10 && < 9.4
template-haskell-2.0 depends on ghc-base >= 9.2 && < 9.10
template-haskell-3.0 depends on ghc-base >= 9.8 && < 9.12
...

Note: should template-haskell be merged into base to make dependencies simpler? [Edit: Teofil Camarasu explained to me that template-haskell has dependencies that base doesn’t have, so that’s probably a no-go]

Status of the “reinstallable base” project

• base has been split in two parts: ghc-internal and base
• wired-in entities have been moved from template-haskell to ghc-internal
• technical changes have been made: removal of wired unit-ids, fix of cabal knowledge about reinstallable packages, hadrian support for 2 versions of template-haskell, etc.
• changes to base interface (exports…) are tracked on CI and aren’t done lightly anymore (they require an accepted CLC proposal)
• I’ve suggested merging ghc-prim, ghc-bignum, ghc-internal into a single package in #24453. Not done yet.
• Probably more. I’m not involved much in this project.

Note that contrary to the model exposed above, there is still a single version of base and template-haskell actively maintained for now.

What about “reinstallable ghc-base”?

We may want to reinstall ghc-base (and the RTS) too! For example to change the bignum backend used to back Integer operations (native vs GMP).

Doing this should be possible. As it’s similar to performing cross-compilation, it’s discussed in the relevant section below.

OC(T) must be linkable with LC(H)

In plain english: the object code produced by the compiler must be loadable/linkable with the running compiler.

This is required by the internal interpreter used to implement:

• Template Haskell
• Annotations
• Plugins
• (GHCi; we don’t really care for this here as it’s not used to compile code)

Here is how the internal interpreter executes code at compilation time:

• GHC compiles the code it wants to run into object code (OC(T))
• it loads/links the object code into its process (e.g. with dlopen when GHC is dynamically linked)
• it lookups the symbol of the function it wants to call in the loaded object code
• it unsafe-coerces the function pointer into a proper Haskell function type
• it calls the function normally

It is safe to unsafe-coerce here because values live in the same heap (the one of the GHC process).

Note: we ignore ByteCode here as the same issue would arise anyway with ByteCode calling into native code (FFI calls, precompiled boot libraries without ByteCode, etc.).

Note: called functions must be pure because if GHC is linked statically with libFOO, then loading some code depending on libFOO will load libFOO again without linking global variables of both libFOO instances together. This was an issue for code depending on the ghc library which used global variables unsafely (now these global variables are properly shared via the RTS).

“OC(T) must be linkable with LC(H)” is a major constraint because it means the object code must have the same ABI as the compiler.

It is however necessary to work around it:

• to build new GHCs
• to support GHC ways
• to support cross-compilation

How do we build a new GHC?

Suppose we want to build GHC-2.0 using the GHC-1.0 program and that they don’t have the same ABI. We use a two-step process as follows:

SC-GHC-2.0 --GHC-1.0(ABI-1.0)--> GHC-2.0(ABI-1.0)

And then:

SC-GHC-2.0 --GHC-2.0(ABI-1.0)--> GHC-2.0(ABI-2.0)

After the two-step process, GHC-2.0(ABI-2.0) has ABI-2.0 and produces object code with ABI-2.0, so it can load it. All good.

Note however that the second step shouldn’t be allowed because of our constraint “OC(T) must be linkable with LC(H)”, i.e. “GHC-2.0(ABI-2.0) must be linkable with GHC-2.0(ABI-1.0)”: the ABIs don’t match! We are only allowed to do this because GHC’s source code doesn’t use any of the features above (TH, annotations, plugins)! This is a constraint but it’s less inconvenient than not being able to modify the ABI ever.

Note: GHC’s newer source code (SC-GHC-2.0) must be compilable by two different GHCs: an old GHC (e.g. GHC-1.0) and the new GHC itself (GHC-2.0). That’s a reason why we are reluctant to add new dependencies to GHC because then we have to adapt them ourselves for them to be usable with the newer GHC. If GHC becomes more stable, then maybe we can start adding more dependencies.

What about cross-compilation and compiler ways?

Remember our illustration:

SC(L) --LC(H)--> OC(T)

With cross-compilation, the compiler host platform H is really different from the object code target platform T. For example, if H is linux-x86-64 and T is javascript then we really can’t link JavaScript “object code” with Linux x86-64 ELF object code.

The same is true for GHC’s “ways”. For example when profiling is enabled, the compiler generates object code with a different ABI. E.g. a heap object for an Int has 1 more field when profiling is enabled so we can’t unsafe-coerce it into an Int expected by a non-profiled compiler without risking segfaults.

External interpreter

This is where the external interpreter is useful. If we have another interpreter process that is built using the target ABI T, then this interpreter can load the object code OC(T) produced by our cross-compiler. The compiler and this external interpreter then just have to exchange properly serialized data (without unsafe coercions) over a pipe or a socket.

1. SC(L) --LC(H)--> OC(T)
2. LC(H) spawns an external interpreter EI(T) for the T target
3. LC(H) asks EI(T) to load OC(T) (and its dependencies)
4. LC(H) asks EI(T) to run function foo and to return its result serialized (e.g. a Template Haskell abstract syntax tree).

Note: step 2 assumes that EI(T) can be spawned on the compiler host platform H. If it isn’t the case, it is possible to spawn a proxy interpreter that communicates with EI(T) spawned on the appropriate platform (e.g. in qemu, wine, or on a remote server). See iserv-proxy.

Who provides the EI(T) external interpreter program?

• GHC distributes some for the ways it has been built to support (profiling, etc.)
• haskell.nix supports even more of them, even automatically using qemu and wine.
• I’ve suggested that they should be built on-demand by GHC (#24731) to avoid the need to distribute them (still to be done).

The external interpreter has a better/safer design than the internal interpreter: why isn’t it the default?

• It’s slower: it needs to spawn another process with its own RTS (GC…) and to serialize data for communication. It’s difficult to compete with the internal interpreter that uses the same process/GC and that (unsafe) coerces values at zero cost.

• Plugins aren’t supported by the external interpreter!

Why aren’t plugins supported by the external interpreter?

In theory, plugins could be supported by the external interpreter: a plugin provides some functions that GHC could call remotely, just as it does for Template Haskell. BUT:

• GHC uses mutable global state (e.g. for the FastString table and for the Unique counter): the external interpreter process would have to query the compiler process every time it wants to read or modify this state. The code would need to be refactored to ensure that a remote command is sent instead of using global variables in the external interpreter itself.

• Plugins have access to the whole compiler state (e.g. via the HscEnv datatype) and it contains mutable variables (MVar)… We would need to be able to serialize these data types to send them back and forth between the compiler and the external interpreter.

A better alternative would be to refine the plugin interface to only allow access to easily serializable types. This requires the design of a proper protocole instead of assuming that GHC internals are fully accessible from a plugin. This would be a massive undertaking that would break every plugin, so it’s unlikely to happen.

Hey, couldn’t we run plugins with the internal interpreter and the rest with the external interpreter?

In theory maybe, in practice no.

A GHC process isn’t multi-target: it assumes that there is a single target platform, a single interpreter, a single loader state, etc.

Package databases are somewhat distinct between packages for plugins and target packages, but not always. Sometimes it’s on purpose (the same packages are expected to be used for the plugins and for the target code) and sometimes not (we reported some occasional leaks between plugins and target code in our paper).

Also, cabal doesn’t provide an interface to declare plugin dependencies separately from target code dependencies (see cabal#2965). So the GHC distinction between plugin packages and target packages is probably never used, except in GHC’s testsuite.

I’ve spent countless hours refactoring GHC’s code to make progress towards supporting several targets in the same process but we’re still far from supporting it. There are many open questions:

• plugins may depend on different wired-in packages than target code. How do we deal with this internally?
• plugins are allowed in the home-unit (the library we’re building): should we build the library twice? (for plugins then for the target)
  • where do we look for the plugin object code in one-shot mode?
• what about multiple home-units?

We’re quickly reinventing cabal inside GHC just for plugins… As I was getting nowhere with the refactoring, I’ve switched to another approach as a workaround: -fplugin-library.

What’s the -fplugin-library workaround?

The idea is to totally bypass GHC’s compilation pipeline for plugins and to load them directly from .so/.dll libraries. It requires two compilers:

• the one used to build plugins (GHC(H,H): runs on H and targets H)
• the one used to generate target code (GHC(H,T): runs on H and targets T).

Both compilers are used like this:

SC-Plugin --GHC(H,H)--> OC-Plugin(H) (e.g. myplugin.so)

SC(L) --{GHC(H,T) -fplugin-library=myplugin.so;... }--> OC(T)

Issues are:

• we need to manage two GHCs; the first one must produce code ABI compatible with the second for plugins
• plugins need to be built manually before calling the GHC process using them.
• plugins loaded with the usual -fplugin flag aren’t automatically supported.

Perhaps GHC could spawn another GHC process to build its plugins and then load them as with -fplugin-library. That would work around the isolation issue without having to refactor the compiler code.

Could we make GHC multi-target instead of using two separate GHCs?

Instead of managing two GHCs or more (one for plugins that will be loaded with -fplugin-library by other GHCs, some more for the targets), we could use a single ghc program to which we pass different flags to select the target.

It is already possible as we’ve made GHC target agnostic: we just have to pass a different “top-dir” with the -B flag to select different settings and boot packages.

We could imagine:

• GHC being aware of the -B flag required to build plugins. Then it would invoke itself to build them with the proper settings/packages.

• a tool with a nice UI that would be used to generate fresh “top-dirs” (settings, boot packages…) for different targets.

The second point would also allow the “reinstallation” of ghc-base mentioned earlier. Using a different ghc-base would be similar to cross-compiling to a different platform: we can’t link the generated code with the compiler because of the discrepancies between the ghc-base used to build the compiler and ghc-base used to build the target code. Similarly, we could “reinstall” all the boot packages: rts, directory, unix, base… Just like what Hadrian does.

Note: plugins depend on the ghc library (ghc-lib), which itself depends on third-party libraries (directory, unix, base…) so they would still be forced to use the libraries used to build GHC itself. This is a real issue. Someone reported exactly this just while I was writing this post:

I am trying to write a plugin but am having the problem where the ghc package want transformers 0.5.6.2 but the package I want the plugin to modify requires 0.6.1.1.

Since, the compiled plugin objects do not intermix with the compiled project objects it seems like this should not actually be a conflict. Is there a way to resolve this?

Plugins/TH status

• -fplugin-library is implemented since 9.6.
• It requires compilers to be stage2, which cross-compilers aren’t by default yet
  • there is some work in progress to fix this: #19174
• Automatic build of the external interpreter is what the JavaScript backend does (approximately)
  • Generalization to other targets is still to do: #24731
• ghc-toolchain is a new tool that should give the nice UI to generate new target “top-dir”
  • I’m not fully aware of its status and roadmap

Ultimately it would be nice to only distribute:

• programs: ghc, ghc-pkg, ghc-toolchain…
• libraries for plugins: ghc-base, ghc-lib…
• settings for plugins

and to let ghc-toolchain (maybe called via cabal) initialize for every target:

• the settings (toolchain to use, etc.)
• the boot packages (ghc-base, base, template-haskell, ghci…)
• an external interpreter

This is my GHC activities report for GHC 9.8

I’ve made roughly 62 commits for the GHC 9.8 series at the time of writing (it’s difficult to know for sure because of the backports).

❯ git merge-base origin/ghc-9.8 origin/ghc-9.6
93ee7e9004c6acf01bf850a9be65c6cfe482e6f0

❯ git log --author "Sylvain Henry" --oneline \
    93ee7e9004c6acf01bf850a9be65c6cfe482e6f0..origin/ghc-9.8 | wc --lines
62

This blog post only lists my personal contributions and thoughts and doesn’t engage my team nor IOG.

JavaScript backend

This time the focus of my team was to make the JavaScript backend more usable. We’ve fixed a lot of issues. I’ve implemented Template-Haskell support by reusing some of the external interpreter code.

Appendix: commit list

# JavaScript backend

a3623053daa JS: implement openat(AT_FDCWD...) (#23697)
bdf2adf7133 JS: implement getMonotonicTime (fix #23687)
b0b88c240d9 JS: support levity-polymorphic datatypes (#22360,#22291)
27df38c2503 JS: better implementation for plusWord64 (#23597)
90846d43179 JS: support -this-unit-id for programs in the linker (#23613)
fb4ecd83171 JS: fix JS stack printing (#23565)
dbb4ad519c3 JS: always recompile when TH is enabled (cf #23013)
a897dc13d2f TH_import_loop is now broken as expected
fab2ad23c9c Fix some recompilation avoidance tests
f84ff161395 Stg: return imported FVs
3249cf1216f Don't use getKey
4d356ea3049 JS: implement TH support
c3a1274c451 JS: don't dump eventlog to stderr by default
9132d529ad2 JS: testsuite: use correct ticket numbers
a5f0c00ee19 JS: factorize SaneDouble into its own module
08d8e9efa24 JS: more triage
15f150c8fcb JS: testsuite: update ticket numbers
4a41ba752fd JS: testsuite: use correct ticket number
f53ac0ae30a JS: fix and enhance non-minimized code generation (#22455)
2972fd66f91 JS: fix getpid (fix #23399)
2f571afe1c2 Fix GHCJS OS platform (fix #23346)
2d5c1ddecf1 Fix remaining issues with bound checking (#23123)
1e9caa1a54e Bump Cabal submodule (#22356)
113e21d7636 Testsuite: replace some js_broken/js_skip predicates with req_c
ab6c1d295cd Testsuite: don't use obsolescent egrep (#22351)
d442ac053f9 JS: fix thread-related primops
eed0d9307b3 GHCi.RemoteTypes: fix doc and avoid unsafeCoerce (#23201)
a8e36892689 JS: fix issues with FD api support
ab9cd52d4fa ghc-heap: remove wrong Addr# coercion (#23181)
f4f1f14f800 ghc-heap: remove wrong Addr# coercion (#23181)
30d45e971d9 Testsuite: use js_skip for T2615 (#22374)
ad765b6f0bb Testsuite: use req_interp predicate for T20214
a1528b68318 Testsuite: use req_interp predicate for T16318 (#22370)
0ab0cc119c6 Testsuite: use appropriate predicate for ManyUbxSums test (#22576)
5b4628aeeca JS: remove dead code for old integer-gmp
239202a2b14 Testsuite: replace some js_skip with req_cmm
8b77f9bfceb JS: fix for overlap with copyMutableByteArray# (#23033)
7825fef9f20 JS: Store CI perf results (fix #22923)
4eb9c234886 JS: make some arithmetic primops faster (#22835)
a203ad854ff Merge libiserv with ghci
324e925be84 JS: disable debugging info for heap objects
df3d94bd295 Testsuite: mark T13167 as fragile for JS (#22921)
c44e5f30cac Testsuite: decrease length001 timeout for JS (#22921)
c1670c6bb0b JS: avoid head/tail and unpackFS
6636b670233 JS: replace "js" architecture with "javascript"
6932cfc798e Fix spurious change from !9568
5640cb1d84d Hadrian: fix doc generation
2fdf22aebda configure: support "windows" as an OS
288fa0179a2 Fix RTS build on Windows
48131ee2d8b Hadrian: fix Windows cross-compilation
e987e345c80 Hadrian: correctly detect AR at-file support
06036d931f5 testsuite: req_smp --> req_target_smp, req_ghc_smp
99757ce8e32 JS: fix support for -outputdir (#22641)


# Misc

db31f25c5ea Add missing int64/word64-to-double/float rules (#23907)
07f858eb1ff Factorize getLinkDeps
b56d737974f NCG: remove useless .align directive (#20758)
1bd32a355bd Factorize hptModulesBelow
4158722a6cf linker: fix linking with aligned sections (#23066)
8af401ccfbe Make WordQuotRem2Op ok-for-speculation too
1148ac723d6 Make Int64/Word64 division ok for speculation too.
4dd02122755 Add quot folding rule (#22152)
59321879419 Add quotRem rules (#22152)

This is my GHC activities report for GHC 9.6

I’ve made roughly 38 commits for the GHC 9.6 series at the time of writing.

❯ git merge-base origin/ghc-9.6 origin/ghc-9.4
27362265e50b59aee9a0ae17560ce091f5501985

❯ git log --author "Sylvain Henry" --oneline \
    27362265e50b59aee9a0ae17560ce091f5501985..origin/ghc-9.6 | wc --lines
38

This blog post only lists my personal contributions and thoughts and doesn’t engage my team nor IOG.

JavaScript backend

This time the focus of my team was to upstream GHCJS (Haskell to GHC compiler) into GHC. The commit adding the JS backend is a squash of a few hundreds of commits (about 522 commits in a branch before the squash) of which I’ve made over 60% (not that commits are a very good metrics).

External static plugins

I’ve spent quite some time trying to make GHC support more than one context at once to support plugins while doing cross-compilation: one context for the target, one context for the host (for plugins). I now think it’s a dead-end because the code still isn’t modular enough. It would be a lot of work to fix and there are opposing forces which make it very unlikely to happen in practice.

I’ve implemented a workaround called “external static plugins”. GHC still doesn’t know how to build plugins in a cross compiler, but if there are built beforehand and put in a library, it can now load them.

Appendix: commit list

# JavaScript

cc25d52e0f6 Add Javascript backend
d86cd08b268 Fix remaining issues with bound checking (#23123)
d5de8a2ab9d Hadrian: enable `-fcheck-prim-bounds` in validate flavour
86a9404a8bc Fix GHCJS OS platform (fix #23346)
5292bdf87f1 JS: fix getpid (fix #23399)
7083db5a7ff JS: fix thread-related primops
b821cdbd911 JS: fix for overlap with copyMutableByteArray# (#23033)
e0cfea59a30 Testsuite: decrease length001 timeout for JS (#22921)
6164ceb89b2 JS: disable debugging info for heap objects
bafa3899758 JS: replace "js" architecture with "javascript"
42ea9cea6ab JS: fix support for -outputdir (#22641)
8b1f1b4503b JS: fix object file name comparison (#22578)

# External static plugins

f95bbdcae3e Add support for external static plugins (#20964)

# Numeric

c284a3e8b1c Bignum: fix right shift of negative BigNat with native backend
9283b7d388d Avoid out-of-bound array access in bigNatIsPowerOf2 (fix #24066)
b6bd8c09e1e Add missing int64/word64-to-double/float rules (#23907)
7f203d00edd Numeric exceptions: replace FFI calls with primops

# Misc

1ae57288cab Fix unusable units and module reexport interaction (#21097)
1838c3d8ed5 GHC.HsToCore.Breakpoints: Slightly improve perf
83467435c4e Avoid using DynFlags in GHC.Linker.Unit (#17957)
f2e707077bf Hadrian: remove unused code
6f5ff4fa0ee Bump hadrian to LTS-19.8 (GHC 9.0.2)
5a5a28dafbb Split GHC.HsToCore.Foreign.Decl
5878f439d42 Enable USE_INLINE_SRT_FIELD on ARM64
acc268064da Some fixes to SRT documentation
913963271f9 MachO linker: fix handling of ARM64_RELOC_SUBTRACTOR
ef3c8d9e077 Don't store LlvmConfig into DynFlags
a4fbb589fd1 STG: only print cost-center if asked to
7a51125dff1 Rts: expose rtsOutOfBoundsAccess symbol
2d7ca624481 linker: fix linking with aligned sections (#23066)
826a311221c configure: support "windows" as an OS
e0d9ef66737 Fix RTS build on Windows
7d86db2be74 Hadrian: fix Windows cross-compilation
ef21780f8b8 Hadrian: correctly detect AR at-file support
b5c714545ab Don't let configure perform trivial substitutions (#21846)
9ab31f42da8 Testsuite: more precise test options
a2f53ac8d96 Add GHC.SysTools.Cpp module
0988a23d211 Enable popcount rewrite rule when cross-compiling

This is my GHC activities report for GHC 9.4

I’ve made roughly 176 commits for the GHC 9.4 series at the time of writing.

❯ git merge-base origin/ghc-9.4 origin/ghc-9.2
05c5c0549bee022be84344cef46f0eded5564c3b

❯ git log --author "Sylvain Henry" --oneline \
    05c5c0549bee022be84344cef46f0eded5564c3b..origin/ghc-9.4 | wc --lines
176

This blog post only lists my personal contributions and thoughts and doesn’t engage my team nor IOG.

Mostly a continuation of my work for GHC-9.2.

Quite a lot of work in progress towards:

• allowing bignum primitives to inline
• supporting plugins in cross-compilers
• Windows support
• performance
• and a few other things.

Appendix: commit list

# Numeric

0d758dc36e1 Add missing int64/word64-to-double/float rules (#23907)
3570eda512d Fix comments about Int64/Word64 primops
c48a7c3a616 Use Word64# primops in Word64 Num instance
be9d78625f1 Fix Int64/Word64's Enum instance fusion
4c86df25d36 Fix Int64ToInt/Word64ToWord rules on 32-bit architectures
e279ea64792 Add missing Int64/Word64 constant-folding rules
c8d89f62a32 Bignum: add missing rule
a901a1ae670 Bignum: allow Integer's signum to inline (#20361)
758e0d7bbe5 Bignum: allow Integer predicates to inline (#20361)
0b575899eca Bignum: constant folding for bigNatCompareWord# (#20361)
4d44058d631 Bignum: transfer NOINLINE from Natural to BigNat
714568bb9e6 Bignum: remove outdated comment
3a5a5c859b5 Bignum: allow naturalToWordClamp/Negate/Signum to inline (#20361)
44886aaba46 Bignum: allow inlining of naturalEq/Ne/Gt/Lt/Ge/Le/Compare (#20361)
e1d02fb0f04 Bignum: allow naturalEq#/Ne# to inline (#20361)
11240b7427c Constant folding for (.&.) maxBound (#20448)
b3267fadd42 Constant folding for negate (#20347)
33eb4a4e396 Constant-folding for timesInt2# (#20374)
d41cfdd4455 Constant folding for ctz/clz/popCnt (#20376)
089de88ef52 Canonicalize bignum literals
f72aa31d36f Bignum: refactor conversion rules
247684adc83 Bignum: remove unused extra files
54880c131db Bignum: fix invalid hs-boot declaration (#19638)
a951e06921f Bignum: add BigNat Eq/Ord instances (#19647)
7bb3443a4fe Fix inlining of division wrappers
c308c9af2e1 Make divModInt# branchless
e50d06752e2 Allow divInt#/modInt# to inline (#18067)
8d069477b77 Refactor modInt# to make it branchless
59cf287d372 Refactor divInt# to make it branchless (#18067, #19636)
df4a0a53691 Bignum: bump to version 1.1 (#19846)
941d3792b2e Rules for sized conversion primops (#19769)
5798357d939 StgToCmm: use correct bounds for switches on sized values
748636388b0 Remove ad-hoc fromIntegral rules
a820f9002d8 Detect underflow in fromIntegral/Int->Natural rule
e5a4cfa5189 Bignum: don't allocate in bignat_mul (#20028)
8838241f7d6 Fix naturalToFloat/Double
453877600b7 Bignum: match on DataCon workers in rules (#19892)
df997fac54b Use quotRemWord in showWord
737b0ae194c Fix Integral instances for Words

# Cross-compilation, build system, Windows support...

69ab7f358a0 configure: support "windows" as an OS
9e07003f2aa Fix RTS build on Windows
35b059d8b52 Hadrian: fix Windows cross-compilation
3247e44aa25 Hadrian: correctly detect AR at-file support
8f8194539b6 linker: fix linking with aligned sections (#23066)
cc59648adbf Hadrian: allow testsuite to run with cross-compilers (#21292)
ed04aed2b2d Hadrian: use IntSet Binary instance for Way (#19209)
c82fb93476b Hadrian: avoid allocations in WayUnit's Read instance (#19209)
c062fac0023 Hadrian: remove useless imports
2509d676a77 Hadrian: avoid allocating in stageString (#19209)
88a945410fe Hadrian: avoid useless allocations in trackArgument
68a3665a73c Hadrian: bump stackage LTS to 18.18 (GHC 8.10.7)
bdeea37efc7 More support for optional home-unit
d9f5490591c Hadrian: fix windows cross-build (#20657)
3302f42a57a Fix windres invocation
8c0aec38c12 Hadrian: fix building/registering of .dll libraries
79a26df1475 Don't expose bignum backend in ghc --info (#20495)
bc498fdfa48 Bignum: expose backendName (#20495)
2800eee24d0 Make Word64 use Word64# on every architecture
20956e5784f Remove target dependent CPP for Word64/Int64 (#11470)
bbb1f6dab34 Hadrian: display command line above errors (#20490)
ecd6d14215e i386: fix codegen of 64-bit comparisons
3d31f11e83a Don't link plugins' units with target code (#20218)
41d6cfc4d36 Add Word64#/Int64# primops
5741caeb045 Only update config.sub when it already exists (#19574)
d014ab0db0c Remove dynamic-by-default (#16782)
8d87975ebe9 Produce constant file atomically (#19684)
b699c4fb0d2 Constants: add a note and fix minor doc glitches
2cdc95f9c06 Don't produce platformConstants file
085983e63bf Read constants header instead of global platformConstants
9c762f27d54 Generate parser for DerivedConstants.h
6623790d748 Hadrian: build check-* with -Wall/-Werror
f0b73ddd8bd Hadrian: add comment to avoid surprises
92257abd4b6 Link with libm dynamically (#19877)
8b9acc4d58f Hadrian: fix .cabal file
323473e86d4 Hadrian: disable profiled RTS with no_profiled_libs flavour transformer
cddafcf6a96 PIC: test for cross-module references
e2d8023dfe1 Add some tests for sized primops
703c0c3c13c Bump libffi submodule to libffi-3.3 (#16940)
82b097b3d43 Remove wired-in names hs-boot check bypass (#19855)
fc9546caf3e genprimopcode: fix bootstrap errors

# Modularity, refactorings

940feaf3c23 Modularize Tidy (#17957)
e6585ca168b Define filterOut with filter
e3f60577b5c Reverse dependency between StgToCmm and Runtime.Heap.Layout
1a7cf0963e3 Avoid redundant imports of GHC.Driver.Session
52ffd38c610 Avoid some SOURCE imports
929c280f991 Derive Enum instances for CCallConv and Safety
ad04953b0d2 Allow hscGenHardCode to not return CgInfos
677c6c913a9 Testsuite: remove arch conditional in T8832
2b1cced1815 NCG: minor code factorization
a31ace56209 Untangled GHC.Types.Id.Make from the driver
7b0c938483b Abstract BangOpts
9728d6c2b62 Give plugins a better interface (#17957)
14e9cab675f Use Monoid in hptSomeThingsBelowUs
b0a1ed5579a Add test for T15547 (#15547)
806e49ae36a Refactor package imports
6fd7da745a5 Remove Indefinite
fa5870d3ac0 Add test for #19641
f6a69fb897b Use an ADT for RecompReason
c662ac7e39a Refactor module dependencies code
da60e6276e4 Fix annoying warning about Data.List unqualified import
628417b42c9 Fix lint issue
a9c0b3ca1d4 Bump haddock submodule
89ee9206548 Use foldGet in getSymbolTable
532c6a54157 Remove UniqSupply from NameCache
599efd90d54 Refactor FinderCache
872a9444df4 Refactor NameCache
d930fecb6d2 Refactor interface loading
751b21448c8 Encapsulate the EPS IORef in a newtype
706fad606db Fix tests
85d7056a53f Move the EPS into UnitEnv
7acfb61777c Move HPT in UnitEnv
0219297c874 Move unit DBs in UnitEnv
29326979eeb Properly initialise UnitEnv
48c2c2af41a Fix genprimopcode warning
491266eeaa6 Make GHC.Runtime.Interpreter independent of GHC.Driver
c7ca3619e25 Interpreter: replace DynFlags with EvalOpts/BCOOpts
8d6b25254e7 Move shift ops out of GHC.Base
67a5a91ef5e Remove useless {-# LANGUAGE CPP #-} pragmas
0ef11907134 Fully remove HsVersions.h
bfabf94f63b Replace CPP assertions with Haskell functions
da56ed41b62 Ensure assert from Control.Exception isn't used
f78c25da058 Move GlobalVar macros into GHC.Utils.GlobalVars
6d712150f8f Dynflags: introduce DiagOpts
58b960d2af0 Make TmpFs independent of DynFlags
c367b39e523 Refactoring module dependencies
4f6726779fa Don't store tmpDir in Settings
6ad25367fbd Fix ASSERTS_ENABLED CPP
2d4cdfda6a7 Avoid unsafePerformIO for getProgName
354ac99de09 Use target platform in guessOutputFile
6f097a8161d Remove useless .hs-boot
b51b4b9777d Make withException use SDocContext instead of DynFlags
61c51c00b6e Fix redundant import
4dc681c7c03 Make Logger independent of DynFlags
9a28680d2e2 Put Unique related global variables in the RTS (#19940)
14956cb89d8 Put tracing functions into their own module
40c0f67fb55 Bump haddock submodule
3a90814fdf5 Parser: make less DynFlags dependent
6b2952cfab2 RTS: fix indentation warning
3e4ef4b2d05 Move warning flag handling into Flags module

# Performance

bc8de322d31 NCG: inline some 64-bit primops on x86/32-bit (#5444)
6fa7591e832 NCG: refactor the way registers are handled
f07b13e38a2 NCG: refactor X86 codegen
90a26f8b0dd Fix documentation about Word64Rep/Int64Rep (#16964)
bc6ba8ef663 Make most shifts branchless
ec26c38bfac Use primOpIds cache more often (#20857)
c080b443142 Perf: use SmallArray for primops' Ids cache (#20857)
9529d859fef Perf: avoid using (replicateM . length) when possible
3f31bfe8c3c Perf: inline exprIsCheapX
352284de379 Perf: remove allocation in writeBlocks and fix comment (#14309)
4c6985cc91b Perf: remove an indirection when fetching the unique mask
4bdafb48b40 Add (++)/literal rule
8b3d98ff376 Don't use FastString for UTF-8 encoding only
0bdb867e37b CoreTidy: handle special cases to preserve more sharing.
3f851bbd473 Enhance pretty-printing perf
1d03d8bef96 Replace (ptext .. sLit) with `text`
e61d2d47c49 Make sized division primops ok-for-spec (#19026)
b128a880b22 Ensure that newtype deriving strategy is used for CTypes
901c79d8de4 Lookup string literals in top-level thunks (fix #16373)
3968cd0c928 Constant-fold unpackAppendCString (fix #20174)
fd7e272eab1 Perf: fix strictness in OccurAnal
de98a0ce8f1 Additional constant-folding rule for binary AND/OR
6eed426bf24 SysTools: make file copy more efficient
23bb09c919a Perf: fix appendFS

# RTS

2929850f013 RTS: open timerfd synchronously (#20618)
5f3991c7cab RTS: try to fix timer races

# Misc

e01ffec2456 ByteCode: avoid out-of-bound read
646c3e21dd3 CI: allow perf-nofib to fail
45a674aacf3 Add `-dsuppress-core-sizes` flag (#20342)
87d93745fa1 Only dump Core stats when requested to do so (#20342)
3fb1afea019 GHCi: don't discard plugins on reload (#20335)
7bff8bf527d Fix pprDeps
92b0037b3e1 Fix recomp021 locale
7bad93a2866 Only create callstack in DEBUG builds
4ae1e53c49f Fix spelling
f192e623f57 Cmm: fix sinking after suspendThread
0f8872ecac4 Fix and slight improvement to datacon worker/wrapper notes

This post is to announce the first public release of a paper we wrote about modularizing GHC (PDF)

As you may know, I’ve been involved into several refactorings of GHC (adding some module hierarchy, ditching DynFlags, etc.). I have been asked a few times to write a document providing an overview of the design that we wanted to reach.

I dutily started to write this document more than two years ago. It started as a compilation of issues about the current design. However it was a bit difficult to state refactoring principles: for example, it seems obvious to me that SDoc mustn’t be coupled with DynFlags, but why? What’s the underlying principle?

Later I found out that I was in fact reinventing Domain-Driven Design (DDD). So I’ve started to reframe the paper as applying DDD principles to GHC. However the paper was put on hold, mostly because welcoming a baby sucked most of my time and energy to work on it.

A few weeks ago, we figured out with John (who had read the draft) that the paper could help a HSoC applicant: https://github.com/haskell-org/summer-of-haskell/issues/143 With the invaluable help of John and Jeff, we strove to make a first release of the paper in less than a week. We obviously failed to meet this deadline. But a few more weeks later, it’s finally there: “Modularizing GHC” paper (PDF) (in case you missed the link above)

It’s an odd paper because it’s a mix of:

• an audit report: a compilation of design issues, with some historical perspectives
• a proposal: a proposed design for GHC, based on DDD
• an experience report: our experience with refactoring GHC, and some method hints for doing it efficiently
• an exposition of DDD to a FP audience, using GHC as an example

Note that in the first draft I planned to also include mentions of Cabal. It was finally left out because it was a highway to despair, and the paper was already very long. We also only briefly mention Hadrian. Both could deserve their own paper of this kind.

This post has been discussed on:

• Reddit: https://www.reddit.com/r/haskell/comments/uhdu4l/modularizing_ghc_paper/
• Discourse: https://discourse.haskell.org/t/modularizing-ghc-paper/4471
• ghc-devs: https://mail.haskell.org/pipermail/ghc-devs/2022-May/020673.html
• HN: https://news.ycombinator.com/item?id=31250141

In this post I discuss the inlining of Integer and Natural operations in Haskell. It’s a promising performance work I’ve been conducting six months ago, which was blocked by an independent issue, but that I will likely resume soon as the issue has been fixed in the meantime.

Note: this post has been first published on IOHK Engineering blog.

To follow this post, you must know that Natural numbers are represented as follows in ghc-bignum:

-- | Natural number
--
-- Invariant: numbers <= WORD_MAXBOUND use the `NS` constructor
data Natural
   = NS !Word#
   | NB !BigNat#

Small naturals are represented with a Word# and large ones with a BigNat# (a ByteArray#).

Now consider the following simple example using Natural:

-- | Add 2 to a Word. Use Natural to avoid Word overflow
foo :: Word -> Natural
foo x = fromIntegral x + 2

There are only small naturals involved: fromIntegral x is small because x is a Word, and 2 is small. We could hope that GHC would use Word# primops to implement this and would allocate a Natural heap object for the result only. However it’s not what happens currently, even in GHC HEAD. In the following STG dump, we can see that a Natural heap object is allocated for x before calling naturalAdd (let bindings in STG reflect heap allocations):

foo1 = NS! [2##];

foo =
    \r [x_sXn]
        case x_sXn of {
        W# x#_sXp ->
        let { sat_sXq = NS! [x#_sXp]; } in  naturalAdd sat_sXq foo1;
        };

Let’s look at naturalAdd:

-- | Add two naturals
naturalAdd :: Natural -> Natural -> Natural
{-# NOINLINE naturalAdd #-}
naturalAdd (NS x) (NB y) = NB (bigNatAddWord# y x)
naturalAdd (NB x) (NS y) = NB (bigNatAddWord# x y)
naturalAdd (NB x) (NB y) = NB (bigNatAdd x y)
naturalAdd (NS x) (NS y) =
   case addWordC# x y of
      (# l,0# #) -> NS l
      (# l,c  #) -> NB (bigNatFromWord2# (int2Word# c) l)

We are clearly in the last case where both arguments are small. It seems beneficial to allow this function to be inlined. If we did we would get:

foo =
    \r [x_s158]
        case x_s158 of {
        W# x#_s15a ->
        case addWordC# [x#_s15a 2##] of {
        (#,#) l_s15c ds_s15d ->
        case ds_s15d<TagProper> of ds1_s15e {
          __DEFAULT ->
              case int2Word# [ds1_s15e] of sat_s15f {
              __DEFAULT ->
              case bigNatFromWord2# sat_s15f l_s15c of ds2_s15g {
              __DEFAULT -> NB [ds2_s15g];
              };
              };
          0# -> NS [l_s15c];
        };
        };
        };

which produces much better assembly code, especially if there is no carry:

    addq $2,%rax       ; add 2 to a machine word
	setc %bl           ; test the carry.
	movzbl %bl,%ebx    ; it could be done
	testq %rbx,%rbx    ; more efficiently
	jne _blk_c17c      ; with "jc"
_blk_c17i:
	movq $NS_con_info,-8(%r12) ; alloc NS datacon value
	movq %rax,(%r12)           ; with the addition result as payload.
	leaq -7(%r12),%rbx         ; make it the first argument
	addq $8,%rbp               ; and then
	jmp *(%rbp)                ; call continuation
...

So why aren’t we always inlining naturalAdd? We even explicitly disallow it with a NOINLINE pragma. The reason is that naturalAdd and friends are involved in constant-folding rules.

For example, consider:

bar :: Natural -> Natural
bar x = x + 2

baz = bar 0x12345678913245678912345679123456798

Currently we get the following Core:

bar1 = NS 2##

bar = \ x_aHU -> naturalAdd x_aHU bar1

baz = NB 99114423092485377935703335253042771879834

You can see that baz is a constant thanks to constant-folding.

However if we let naturalAdd inline we get:

baz
  = case bigNatAddWord# 99114423092485377935703335253042771879832 2##
    of ds_d11H
    { __DEFAULT ->
    NB ds_d11H
    }

baz is no longer a constant.

A solution would be to add constant-folding rules for BigNat# functions, such as bigNatAddWord#. This is exactly what we have started doing in #20361. Our new plan is:

• Make BigNat# operation NOINLINE and add constant-folding rules for them
• Make Integer/Natural operations INLINEABLE (expose their unfolding)
• Hence rely on constant-folding for Word#/Int#/BigNat# to provide constant folding for Integer and Natural

The good consequences of this plan are:

• Less allocations when bignum operations are inlined and some of the arguments are known to be small/big or fully known (constant).
• Integer and Natural are less magical: you can implement your own similar types and expect the same performance without having to add new rewrite rules

There were some unforeseen difficulties with this plan though:

1. Some of the rewrite rules we need involve unboxed values such as BigNat# and Word# and the weren’t supported. Luckily, this has been recently fixed (#19313) by removing the “app invariant” (#20554). Thanks Joachim! That’s the reason why we could resume this work now.
2. Some unfoldings (RHSs) become bigger due to the inlining of bignum operations. Hence they may not themselves be inlined further due to inlining thresholds even if it would be beneficial. A better inlining heuristic would fix this (see #20516). It will likely be the topic of the next post.

This is my GHC activities report for GHC 9.2.

I’ve made roughly 204 commits for the GHC 9.2 series at the time of writing.

❯ git merge-base origin/ghc-9.2 origin/ghc-9.0
19e80b9af252eee760dc047765a9930ef00067ec

❯ git log --author "Sylvain Henry" --oneline \
    19e80b9af252eee760dc047765a9930ef00067ec..origin/ghc-9.2 | wc --lines
204 

This blog post only lists my personal contributions and thoughts and doesn’t engage my team nor IOG.

Constant-folding

fromIntegral is the go-to function to convert between integral types. It is implemented as follows:

fromIntegral :: (Integral a, Num b) => a -> b
fromIntegral = fromInteger . toInteger

It used to be a bit magical because it had a lot of associated rewrite rules to avoid passing through Integer when possible. For example, fromIntegral :: Int -> Word can be implemented much more efficiently by using the int2Word# primop (which is a no-op in assembly). This was done with a rewrite rule on fromIntegral.

Now, all the rewrite rules for fromIntegral are gone as the rewriting is done on Integer primitives themselves. For example, integerToWord# (integerFromInt# x) ~~> int2Word# x.

It is especially useful for alternatives Prelude that can now provide safer numeric conversion functions without relying on fromIntegral. Also as fromIntegral was (and still is) inlined to avoid function calls, we are more likely to catch constant-folding occuring after some inlining happened, as well as constant-folding occuring from direct use of toInteger/fromInteger or underlying primitives.

Note that Natural is also taken into account everytime I’m changing something Integer related to avoid differences between the two bignum types.

64-bit primops

GHC provides a few primitive types: Int#, Word# but also Int8#, Word32#, etc.. Word64# and Int64# were only used on 32-bit architectures (on 64-bit architectures Int#/Word# are 64-bit and can be used instead) but we are working (with John Ericson) on making them available on 64-bit architectures too. To make GHC a multi-target compiler, we want to avoid having primops whose type depend on the target so we want to use the explicitly sized Int64#/Word64# types for them.

At this point it is only teasing for the GHC 9.4 activities report. But in the process, in GHC 9.2 we have made all the operations on Int64#/Word64# proper primops instead of FFI calls. It allowed us to add constant-folding for these operations! There were some noticeable performance improvements on 32-bit architectures:

T9203(normal) run/alloc    89870808.0    66662456.0 -25.8% GOOD

haddock.Cabal(normal) run/alloc 14215777340.8 12780374172.0 -10.1% GOOD

haddock.base(normal) run/alloc 15420020877.6 13643834480.0 -11.5% GOOD

Target constants

GHC doesn’t fully control code generation by the C toolchain used to build the runtime system (RTS). As such, GHC’s build system queries some structure sizes, field offsets and other values from object files generated by the C toolchain (this is the role of utils/deriveConstants program). These constants are then used when compiling Haskell code, e.g. the native code generator uses them to generate assembly.

These constants were stored in a globally installed platformConstants file. As we want to make GHC multi-target and the RTS reinstallable, it had to change. For example the reinstalled RTS may slightly change the structure representations (e.g. have packed fields, some field order changes, some alignment changes…). Hence platform constants are now only stored in the RTS unit, not globally.

Windows

At some point GHC’s Windows CI capacity became too small and it started slowing the merge process too much. So I’ve decided to rent a Windows machine and to install a Gitlab runner on it to build GHC…

I’ve discovered that Windows is a very hostile (Haskell) development platform. For example Git automatically converts LF line-ending characters into CRLF on checkout by default leading to weird build failures.

It seems I was the first one using a newer Cabal version which failed building GHC because a Windows-only utility (touchy) had a wrong cabal-version in its .cabal file.

More importantly it seems that I was also the first one building GHC on Windows with autotools 2.70 (first new version after 8 years) which broke the support of auxiliary files (e.g. config.guess) on Windows (https://gitlab.haskell.org/ghc/ghc/-/issues/19189). The handling of paths and path separators is a complete mess (see https://github.com/haskell/cabal/issues/7494 and https://github.com/haskell/cabal/issues/7649 where cabal devs try to fix the issue properly) so it was simpler to fix the issue by removing the use of config.guess and config.sub. We support 3.5 OSs and 4 architectures, why do we have to use these scripts anyway?

Modularity

A lot of time has been spent refactoring the compiler. I’ve started writing a paper/report about it so I won’t go into much details here.

Error as ADTs

I’ve kicked off the use of ADTs to represent errors and warnings so that they are easier to consume by other tools (instead of forcing them to parse text messages). This [first patch](https://gitlab.haskell.org/ghc/ghc/-/commit/a5aaceecaa) fixed the parser.

For some reason this patch improves performance:

Surprisingly, this patch enhances performance. On CI
(x86_64/deb9/hadrian, ghc/alloc):

   parsing001         -11.5%
   T13719             -2.7%
   MultiLayerModules  -3.5%
   Naperian           -3.1%

Appendix: commit list

# numeric

4d2bbd676e Remove ad-hoc fromIntegral rules
a24044dcb0 Ensure that newtype deriving strategy is used for CTypes
ab14cdbecb Fix naturalToFloat/Double
d1a9167606 Additional constant-folding rule for binary AND/OR
e552296414 Bignum: bump to version 1.1 (#19846)
1d7840f509 Detect underflow in fromIntegral/Int->Natural rule
882cc36382 Bignum: add BigNat Eq/Ord instances (#19647)
d197fb3d5c Bignum: remove unused extra files
3dc0815eea Bignum: fix invalid hs-boot declaration (#19638)
e594d067b2 Bignum: match on DataCon workers in rules (#19892)
773e2828fd Bignum: add Natural constant folding rules (#15821)
5e71dd3379 Bignum: fix bogus rewrite rule (#19345)
6f9a817fff Bignum: fix for Integer/Natural Ord instances
bff74de713 Bignum: make GMP's bignat_add not recursive
74ee1237bf Bignum: fix bigNatCompareWord# bug (#18813)
12191a99d3 Bignum: match on small Integer/Natural
8dd4f40512 Bignum: implement integerPowMod (#18427)
12c06927a0 Bignum: implement integerRecipMod (#18427)
e5523324d2 Bignum: add integerNegate RULE
1cde295c54 Bignum: add bigNatFromWordArray
04bc50b3c8 Bignum: implement extended GCD (#18427)
92daad241b Bignum: refactor backend modules
b689f3db02 Bignum: add clamping naturalToWord (fix #18697)
85e6212349 Bignum: add missing compat import/export functions
a8a2568b7b Bignum: add BigNat compat functions (#18613)
dc476a5040 Bignum: fix BigNat subtraction (#18604)
3f2f771869 Bignum: add more BigNat compat functions in integer-gmp
d613ed7624 Bignum: add backward compat integer-gmp functions
380638a336 Bignum: fix powMod for gmp backend (#18515)
6ee07b494d Bignum: add support for negative shifts (fix #18499)
96c31ea1f0 Fix bug in Natural multiplication (fix #18509)
0a3723876c Fix documentation and fix "check" bignum backend (#18604)
2754419673 Natural: fix left shift of 0 (fix #19170)
730ef38f46 Simplify constant-folding (#18032)
a98593f0c7 Refactor numeric constant folding rules
ffc3da474b Remove outdated note
85d899c8d3 Make proper fixed-width number literals

# cross-compilation / multi-target

52e5208367 Add Word64#/Int64# primops
ffb4762f2c Constants: add a note and fix minor doc glitches
201e29c74e Don't produce platformConstants file
0e9069be0e Read constants header instead of global platformConstants
ddd8fe23fd Generate parser for DerivedConstants.h
6c8702aede Remove dynamic-by-default (#16782)
a91dcb669b Don't get host RTS ways via settings (#18651)
accdb24a08 Expose RTS-only ways (#18651)
b695e7d736 Rename ghci flag into internal-interpreter
7721b923d5 Move GHC.Platform into the compiler
73145d57f9 Remove dead code in utils/derivConstants
9dfeca6c20 Remove platform constant wrappers
6333d73910 Put PlatformConstants into Platform
f7cc431341 Replace HscTarget with Backend
735f9d6bac Replace ghcWithNativeCodeGen with a proper Backend datatype
e079bb721e Correctly test active backend
8ea33edb2b Remove unused sGhcWithNativeCodeGen
bab4ec8f62 Don't panic if the NCG isn't built (it is always built)
0f17b930c1 Remove unused "ncg" flag
cdbd16f545 Fix toArgRep to support 64-bit reps on all systems
05c5c0549b Move loader state into Interp
d8dc0f9623 Fix array and cleanup conversion primops (#19026)
94bbc45df3 Use target Int/Word when detecting literal overflows (#17336)
2895fa6035 ghci: reuse Arch from ghc-boot
37f0434d65 Constant-folding: don't pass through GHC's Int/Word (fix #11704)

# Windows

dd23bd740f Windows: fix crlf on checkout
bcaa36c427 Fix Windows build with autoconf >=2.70 (#19189)
5b752b1d5c touchy: use a valid cabal-version
42ab06f793 Replace more autotools obsolete macros (#19189)
0f2891f018 configure: avoid empty lines in AC_CONFIG_FILES

# Modularity / Refactorings

d0e8c10d58 Move Unit related fields from DynFlags to HscEnv
76be0e32d6 Move SizedSeq into ghc-boot
690c894616 Parser: move parser utils into their own module
1ff61314fa Fix wrong comment about UnitState
c85f4928d4 Refactor -dynamic-too handling
72f2257c79 Don't initialize plugins in the Core2Core pipeline
ecfd0278cb Move Plugins into HscEnv (#17957)
14ce454f72 Linker: reorganize linker related code
08e6993a1b Move loadDecl into IfaceToCore
0e9f6defbd Split GHC.Driver.Types
9648d680b4 Remove pdocPrec
0a5f291859 Parser: don't require the HomeUnitId
9bbc84d20d DynFlags: refactor DmdAnal
6a243e9daa Cache HomeUnit in HscEnv (#17957)
386c2d7ff8 Use UnitId in the backend instead of Unit
93d5de165a Don't import GHC.Unit to reduce the number of dependencies
dafe79433c Parser: remove some unused imports
a946c7ef9e Less DynFlags in Header parsing
a5aaceecaa Use ADTs for parser errors/warnings
9befd94d79 Remove unused global variables
df3f588075 Remove unsafeGlobalDynFlags (#17957, #14597)
8e3f00dd24 Make the parser module less dependent on DynFlags
a997fa01d9 Preliminary work towards removing DynFlags -> Driver.Ppr dependency
a584366b1d Remove sdocWithDynFlags (fix #10143)
667d63558a Refactor CLabel pretty-printing
7f2785f2d6 Remove pprPrec from Outputable (unused)
37aa224ad5 Add note about OutputableP
e45c85446d Generalize OutputableP
ca48076ae8 Introduce OutputableP
8af954d202 Make ghc-boot reexport modules from ghc-boot-th
ffae579211 Add comments about sm_dflags and simpleOptExpr
b3df72a699 DynFlags: add sm_pre_inline field into SimplMode (#17957)
3f32a9c0f4 DynFlags: add UnfoldingOpts and SimpleOpts
8c89268905 DynFlags: add OptCoercionOpts
df04b81e12 Move DynFlags test into updateModDetailsIdInfos's caller (#17957)
220ad8d67a DynFlags: don't pass DynFlags to cmmImplementSwitchPlans
89ce7cdf97 DynFlags: use Platform in foldRegs*
1d6d648866 Don't rely on CLabel's Outputable instance in CmmToC
bcb68a3f7f Don't store HomeUnit in UnitConfig
0b17fa185a Refactor UnitId pretty-printing
659eb31b7a NCG: Dwarf configuration
50eb4460cd Don't use DynFlags in CmmToAsm.BlockLayout (#17957)
293c7fba6c Put CFG weights into their own module (#17957)
0c5ed5c7eb DynFlags: refactor GHC.CmmToAsm (#17957, #10143)
accbc242e5 DynFlags: disentangle Outputable
6334815540 Use a type alias for Ways
3b15dc3cfb DynFlags: don't use sdocWithDynFlags in GHC.CmmToAsm.Dwarf.Types
56a7c19337 Refactor CLabel pretty-printing
175cb5b404 DynFlags: don't use sdocWithDynFlags in datacon ppr
342a01af62 Add GHC.Platform.Profile
15b2b44fe3 Rename GHC.Driver.Ways into GHC.Platform.Ways
e443846ba0 DynFlags: store printer in TraceBinIfaceReading
7f44df1ec6 Minor refactoring of Unit display
6e2db34bdf Add accessors to ArchOS
ffc0d578ea Add HomeUnit type
8e2f85f6b4 Refactor Logger
e9d0dd4554 Move showSDocUnsafe/showPprUnsafe back into GHC.Utils.Outputable
daa6363f49 DynFlags: move temp file management into HscEnv (#17957)
3aceea907f Don't pass homeUnitId at ExternalPackageState creation time (#10827)
957b53760e Core: introduce Alt/AnnAlt/IfaceAlt datatypes
fd0945b7bf Move Hooks into HscEnv

# performance

e393f213f5 Allow fusion with catMaybes (#18574)
05550a5abc Avoid roundtrip through SDoc
b1eb38a0a7 Perf: make SDoc monad one-shot (#18202)
8a51b2ab74 Make IOEnv monad one-shot (#18202)
bee43aca82 Rewrite and move the monad-state hack note
77398b678a Avoid allocations in `splitAtList` (#18535)
acf537f9fe Make splitAtList strict in its arguments
fe4202cef7 Always INLINE ($!)
b677d09344 Make sized division primops ok-for-spec (#19026)
87ae062ab8 Compute length only once in foldBal
d412cd108b Write explicit IOEnv's Functor and MonadIO instances (#18202)
8a433a3c88 Fix leaks of the HscEnv with quick flavour (#19356)
72c0e07869 Make known names simple ConApps (#19386)
887eb6ec23 Enhance Data instance generation
ae8379abb8 Ppr: compute length of string literals at compile time (#19266)
0249974e76 Fix strictness in TyCo.Tidy (#14738)
5ae73f69e5 Add regression test for #16577
6c771aafa3 Implement Unique supply with Addr# atomic primop
4af6126d17 Use static array in zeroCount

# build system / CI

a3129aa743 Hadrian: fix .cabal file
31ee48dc64 CI: reduce xz compression for non release/nightly jobs
18e106a8df Add missing .hi-boot dependencies with ghc -M (#14482)
9c87f97ee9 Fix spurious failures of T16916 on CI (#16966)
e516ef7eb7 Hadrian: fix flavour parser
bd877edd94 Hadrian: show default ghc-bignum backend (fix #18912)
659fcb1493 Fix project version for ProjectVersionMunged (fix #19058)
7cb5df9617 hadrian: fix ghc-pkg uses (#17601)
75fc1ed58b Hadrian: fix detection of ghc-pkg for cross-compilers
66414bdf40 configure: fix the use of some obsolete macros (#19189)
4813486f87 Move Hadrian's wiki pages in tree (fix #16165)
3486ebe6f9 Hadrian: don't fail if ghc-tarballs dir doesn't exist
b4a929a1e5 Hadrian: fix libffi tarball parsing
aaa8f00fa0 Validate script: fix configure command when using stack
db236ffc03 Testsuite: increase timeout for T18223 (#18795)
3a16d764f3 ghci: don't compile unneeded modules
e691a5a046 Hadrian: add quick-debug flavour
f1accd0096 Add quick-validate Hadrian flavour (quick + -Werror)
2da933084b Hadrian: fix slow-validate flavour (#18586)
459afeb592 Fix build systems
b3143f5a08 Enhance metrics output
9c1b8ad931 Bump Stack resolver
a01e78cc5b Don't build extra object with -no-hs-main
226cefd0a8 Fix fake import in GHC.Exception.Type boot module

# misc

644424afc5 GHCi: don't discard plugins on reload (#20335)
3a81545208 RTS: try to fix timer races
6b10163e0d Disable bogus assertion (#19489)
d14a20686c Enhance pass result forcing
3f9af89128 Add a flag to dump the FastString table
38748d5fb4 Minor simplification for leak indicators
fe78997845 IntVar: fix allocation size
a3d995fa18 Fix -dynamic-too with wired-in modules (#19264)
13d876bab3 Enhance nested TransCo pretty-printing
db16302cfd LLVM: fix sized shift primops (#19215)
29173f8888 Factorize and document binder collect functions
01ea56a22d Arrows: collect evidence binders
84dcb8440e Revert "Remove SpecConstrAnnotation (#13681)" (#19168)
fe344da9be Missing fields: enhance error messages (#18869)
d930687a07 Show missing field types (#18869)
f59c34b8e4 Support package qualifier in Prelude import
480a38d4ad rts: don't use siginterrupt (#19019)
f9f9f030d7 Arrows: correctly query arrow methods (#17423)
381eb66012 Display FFI labels (fix #18539)
52114fa0f9 Add Addr# atomic primops (#17751)
4c407f6e71 Export SPEC from GHC.Exts (#13681)
81560981fd Don't use LEA with 8-bit registers (#18614)
22f5d9a951 GC: Avoid data race (#18717, #17964)
809f09e8a7 Fix parsing of PIE flags
0e8b923d3c Apply suggestion to compiler/GHC/SysTools.hs
74d4017b74 Fix -flink-rts (#18651)
0bb02873ee Add test for T18574
4b4fbc58d3 Remove "Ord FastString" instance
884245dd29 Fix FastString lexicographic ordering (fix #18562)
2d8ca91703 Fix -ddump-stg flag
7c274cd530 Fix minimal imports dump for boot files (fix #18497)
9e2930c3e1 Bump CountParserDeps
ff1b7710c9 Add test for #18064

Changelog

• 2021-03-15: discussed on reddit

This is my GHC activities report for GHC 9.0.

If I’ve got the following script right, I’ve made 225 commits for the GHC 9.0 series at the time of writing.

❯ git merge-base origin/ghc-9.0 origin/ghc-8.10
098d50176a2521d08efe6fdbab0ede2b7419786a

❯ git log --author "Sylvain Henry" --oneline \
    098d50176a2521d08efe6fdbab0ede2b7419786a..origin/ghc-9.0 | wc --lines
225

Module hierarchy

I have completed the renaming of every GHC module to introduce a hierarchy.

Haddock for the GHC library before and after the changes:

You might think: “So what? You only renamed a few modules”. But in fact it wasn’t that simple as I had neglected the amount of discussion that this kind of change would trigger.

First attempt: I did the work once in June 2017 in a single big patch that was very difficult to review and to rebase and consequently that has never been merged. I have been asked to write a ghc-proposal which has then been flagged out-of-scope (quite demoralizingly). I have implemented ghc-api-compat, a package to help compiling codes using the old naming (sadly the latest version can’t be uploaded on Hackage because of Cabal bug #4629). So I finally gave up in December 2017.

After enough time had passed to forget about this, I made a second attempt in 2019. This time with much smaller patches and waiting for one patch to be merged before starting working on the other to avoid too much rebasing work. As the process took longer than expected (it ended in 2020), the refactoring wasn’t completed for the 8.10 release. That’s why 8.10 has only some hierarchy (e.g. GHC.Hs and GHC.StgToCmm).

The whole story is tracked in #13009.

This change mostly matters if you are a GHC developer or a user of the GHC API because that’s what you interact directly with. But doing this was only a first step before starting to modularise GHC’s codebase. The latter is still a lot of (functional) spaghetti code but it’s getting better. Now we can try to enforce some rules such as “no, every part of the compiler shouldn’t directly use the driver state (HscEnv)”, etc.

Thanks to Takenobu Tani for fixing many references after these changes (in code comments, in the Wiki, etc.). Thanks to Richard Eisenberg and Simon Peyton Jones for their support.

Users guide

I found GHC’s users guide quite difficult to navigate especially because of its theme: the table of contents on the right was often useless because we had to scroll up a lot to find it. So I’ve switched to the now very common ReadTheDocs theme which has the TOC always visible on the left.

• GHC 8.10: https://downloads.haskell.org/ghc/8.10.1/docs/html/users_guide/index.html
• GHC 9.0.1: https://downloads.haskell.org/ghc/9.0.1/docs/html/users_guide/index.html

I had to adapt the CSS of the theme to try to match the old theme in some cases. It’s far from being perfect so anyone with CSS skills should feel free to enhance it.

While doing this change I noticed that the GHC Language Features page was huge and slow to render. So I’ve done the tedious work of splitting it. By doing this I fixed issue #17316 that I had missed before and where other devs had been discussing doing this. It looks better now but I have done the split my own way (I don’t even remember exactly which criteria I’ve followed, it was more than a year ago). So don’t hesitate to propose changes to enhance this.

ghc-bignum

In August 2019 I started toying with an implementation of Natural numbers. When I started working for IOHK in September 2019 it turned out it would be useful to replace the previous Haskell native implementation (integer-simple) with a faster one for cross-compilation concerns. So I worked on it and ghc-bignum was merged in June 2020.

We published a post about this work in July 2020 on IOHK blog: https://iohk.io/en/blog/posts/2020/07/28/improving-haskells-big-numbers-support/

I have implemented only simple algorithms (division was tricky enough to implement) but performance is pretty good, especially compared to integer-simple. The following chart from the blog post linked above compares performance results of some basic operations (note the log scale):

Providing a new native Haskell implementation of Integer/Natural was only part of the project. ghc-bignum also refactors the way support for big numbers is implemented. Instead of two separate packages (integer-gmp or integer-simple), it’s now always ghc-bignum but built with different Cabal flags to select the backend. This simple change allows other packages to depend on ghc-bignum unconditionally starting with GHC 9.0.

Note that a integer-gmp package is still distributed for backward compatibility, but it relies on ghc-bignum.

Implementing ghc-bignum led me to learn about and to modify both build systems of GHC, something that you don’t want to have to do. It also uncovered tricky issues like raising exceptions defined in base package from a package that base depends on…

Thanks a lot to early testers who uncovered bugs in ghc-bignum! Implementing this library led me to learn more about program proofs. A stupid error in such library could easily go unnoticed and then any user of such numeric code could compute some wrong results: it’s quite scary!

By the way, you shouldn’t use GHC 9.0.1 in production because of a silly bug I’ve introduced. My patch implementing constant folding for Natural numbers has been merged only in the final release of GHC 9.0.1 and sadly it introduced a bogus rule. Constant folding rules in GHC are the second reason for me to learn about program transformation proofs.

Modularisation

Finally most of my time has been spent in modularising the GHC codebase. Or more precisely, disentangling the codebase (modularisation should come later).

If you have ever used the ghc-api or looked at the codebase, your first reaction has probably been “WTF?!”. In particular, you must have encountered the most infamous datatype in the compiler: DynFlags. Despite its name, it contains a lot of random stuff. My guess is that as there was always a DynFlags in scope, it ended up being used as a Reader monad state with some IORefs in it. And when it wasn’t in scope… well it always was because there was a unsafeGlobalDynFlags global variable too.

As we were (and still are) working towards making GHC multi-target (currently a ghc executable can only produce code for a single target, e.g. x86-64-linux), it became necessary to untangle all this.

For example, before, the compiler could assume that there was only a single platform: the target one. So it just had to query DynFlags.targetPlatform everywhere. Now we have to pass the platform as a parameter to the appropriate functions. That’s why many functions that used to take a DynFlags parameter now take a Platform parameter. See this commit for an example of such huge patches.

As my current objective is to implement support for compiler plugins (i.e. Haskell code for the host) in cross-compilers (i.e. GHC compiler producing codes for a target different of the host, e.g. JavaScript), I have also made a lot of changes to the code handling units (compiled packages).

#17957 is the ticket that I use to track generic refactorings. #14335 is more specific to compiler plugins. There is still a lot of work to do in both cases.

Thanks to John Ericson who has made and still makes a lot of similar refactorings.

Conclusion

If you are a GHC API user, there is 100% chance that I contributed to break your code with this release. Hopefully it’s for the best in the future so please don’t be too mad!

Don’t forget to wait for GHC 9.0.2 before using GHC 9.0 in production.

I have done a few other random things; see the commit list below.

Appendix: commit list

# ghc-bignum

f5e5a1febf Bignum: fix bogus rewrite rule (#19345)
ba089952f0 Bignum: add Natural constant folding rules (#15821)
f4b11d23fe Bignum: add clamping naturalToWord (fix #18697)
c6851770e7 Bignum: fix for Integer/Natural Ord instances
bba8f79c7e Bignum: make GMP's bignat_add not recursive
d09e7e41cf Bignum: fix bigNatCompareWord# bug (#18813)
a740aa0bfc Bignum: match on small Integer/Natural
5d414fdc01 Bignum: implement integerPowMod (#18427)
175d714126 Bignum: implement integerRecipMod (#18427)
89a001505a Bignum: add integerNegate RULE
ebcc09687b Bignum: add bigNatFromWordArray
74f3f581dd Bignum: implement extended GCD (#18427)
6c98a930ee Bignum: refactor backend modules
c23275f4dd Bignum: add missing compat import/export functions
d5c3a027ec Bignum: add BigNat compat functions (#18613)
bf8bb9e785 Bignum: fix BigNat subtraction (#18604)
3745bdb69b Bignum: add more BigNat compat functions in integer-gmp
eab2511ed2 Bignum: add backward compat integer-gmp functions
817f94f51a Bignum: fix powMod for gmp backend (#18515)
b4cccab3cc Fix bug in Natural multiplication (fix #18509)
324967891a Bignum: add support for negative shifts (fix #18499)
929d26db30 Bignum: don't build ghc-bignum with stage0
d3bd689784 BigNum: rename BigNat types
1b3d13b68c Fix ghc-bignum exceptions
bccf3351a2 Add ghc-bignum to 8.12 release notes
a403eb917b ghc-bignum: fix division by zero (#18359)
9f96bc127d ghc-bignum library
fa4281d672 Bump bytestring and text submodules
dceecb093c Update Hadrian
f817d816e6 Update testsuite
aa9e7b7196 Update `make` based build system
0f67e3447e Update `base` package
96aa57878f Update compiler
57db91d8ee Remove integer-simple
f82a2f90ce Document GMP build [skip ci]
a3e907630f Fix documentation and fix "check" bignum backend (#18604)
a589fb949c Natural: fix left shift of 0 (fix #19170)
558d4d4a2b Split integerGmpInternals test in several parts
581753790d Hadrian: minor GMP refactoring

# Build system

faa36e5b36 Hadrian: ignore in-tree GMP objects with ``--lint``
c3c801e3c3 Replace more autotools obsolete macros (#19189)
062b3a7e8b touchy: use a valid cabal-version
bea3df2e85 Fix Windows build with autoconf >=2.70 (#19189)
0789f7a18a Hadrian: show default ghc-bignum backend (fix #18912)
d1de5c2271 Use Hadrian by default in validate script (#17527)
d25b6851bb Hadrian: ghc-gmp.h shouldn't be a compiler dependency
23e4e04700 Hadrian: fix PowerPC64le support (#17601)
b420fb2474 Hadrian: fix hp2ps error during cross-compilation
15ccca16e2 Hadrian: fix distDir per stage
7a07aa7181 Hadrian: fix cross-compiler build (#16051)
95da76c2b9 Hadrian: fix binary-dist target for cross-compilation
85fc32f03a Hadrian: fix dyn_o/dyn_hi rule (#17534)
75a185dc2a Hadrian: fix --summary
729bcb0271 Hadrian: fix build on Mac OS Catalina (#17798)
b989845e37 Hadrian: fix absolute buildroot support (#17822)
ee2c50cbee Hadrian: track missing configure results
2a2f51d79f Use configure script to detect that we should use in-tree GMP on Windows
34c7d23074 Fix Hadrian's ``--configure`` (fix #17883)
9d09411122 Hadrian: `docs` rule needs `configure` (#17840)
e2cce99732 Hadrian: fix source-dist target (#17849)
3cea67955a Make: fix sdist target (#17848)
d7029cc09e Hadrian: refactor GMP in-tree build support (#17756)
7550417ac8 Hadrian: drop Sphinx flag checking for PDF documentation (#17825)
9f2c3677b3 GMP expects the Target platform as --host parameter
1bfd825943 Ensure that Hadrian is built correctly before using it
414e2f6263 Force -fPIC for intree GMP (fix #17799)
4f11713567 Make: refactor GMP rules

# Numeric stuff (preliminaries to ghc-bignum)

fdcc53babb Optimise genericIntMul2Op
7a51b587ad Add constant folding rule (#16402)
35afe4f3b1 Use Int# primops in `Bits Int{8,16,32,64}` instances
fbbe18a274 Use the new timesInt2# primop in integer-gmp (#9431)
5f7cb423d7 Add `timesInt2#` primop
3656dff825 LLVM: fix MO_S_Mul2 support (#18434)
b5768cce02 Don't use timesInt2# with GHC < 8.11 (fix #18358)
1cca12edaa Constant-folding: don't pass through GHC's Int/Word (fix #11704)
aba51b6586 Add arithmetic exception primops (#14664)

# Module hierarchy

20800b9a9e Split GHC.Iface.Utils module
a426abb9b4 Rename GHC.Hs.Types into GHC.Hs.Type
37430251c3 Rename GHC.Core.Arity into GHC.Core.Opt.Arity
528df8ecb4 Modules: Core operations (#13009)
18a346a4b5 Modules: Core (#13009)
1941ef4f05 Modules: Types (#13009)
255418da5d Modules: type-checker (#13009)
15312bbb53 Modules (#13009)
af33244212 Modules: Utils and Data (#13009)
1500f0898e Modules: Llvm (#13009)
240f5bf6f5 Modules: Driver (#13009)
817f93eac4 Modules: Core (#13009)
1b1067d14b Modules: CmmToAsm (#13009)
cf739945b8 Module hierarchy: HsToCore (cf #13009)
da7f74797e Module hierarchy: ByteCode and Runtime (cf #13009)
6e2d9ee25b Module hierarchy: Cmm (cf #13009)
d491a6795d Module hierarchy: Renamer (cf #13009)
99a9f51bf8 Module hierarchy: Iface (cf #13009)
eb6082358c Module hierarchy (#13009): Stg

# Users guide

0a5e4f5f7d Split glasgow_exts into several files (#17316)
aeea92ef6f Switch to ReadTheDocs theme for the user-guide

# Modularisation

ce5408c062 Replace ghcWithNativeCodeGen with a proper Backend datatype
666acbd43d Correctly test active backend
e27698ce0b Remove unused sGhcWithNativeCodeGen
dbf77b79d9 Remove unused "ncg" flag
4e22de2a6f Don't panic if the NCG isn't built (it is always built)
30caeee751 DynFlags: remove use of sdocWithDynFlags from GHC.Stg.* (#17957)
0ddae2ba97 DynFlags: factor out pprUnitId from "Outputable UnitId" instance
41d2649288 DynFlags: avoid the use of sdocWithDynFlags in GHC.Core.Rules (#17957)
f08d6316d3 Replace Opt_SccProfilingOn flag with sccProfilingEnabled helper function
3cdd8d69f5 NCG: correctly handle addresses with huge offsets (#15570)
a04020b88d DynFlags: don't store buildTag
7ad4085c22 Fix invalid printf format
cad62ef119 Add tests for #17920
5f6a066551 LLVM: refactor and comment register padding code (#17920)
2636794d1a CmmToC: don't add extern decl to parsed Cmm data
7750bd456f Cmm: introduce SAVE_REGS/RESTORE_REGS
d4a0be7580 Move tablesNextToCode field into Platform
2af0ec9059 DynFlags: store default depth in SDocContext (#17957)
eb8115a8c4 Move CLabel assertions into smart constructors (#17957)
456e17f035 Bump haddock submodule and allow metric decrease
c10ff7e7e5 Doc: fix some comments
1fbb4bf5f3 NCGConfig: remove useless ncgUnitId field
bfd0a78cdd Don't return preload units when we set DyNFlags
ac964c8350 Put database cache in UnitConfig
28d804e1e1 Create helper upd_wired_in_home_instantiations
4274688a63 Move distrustAll into mkUnitState
fca2d25ff7 DynFlags: add UnitConfig datatype
8408d521a6 DynFlags: merge_databases
a444d01bc9 DynFlags: reportCycles, reportUnusable
42c054f6cd DynFlags: findWiredInUnits
4b53aac1e2 Refactor and document closeUnitDeps
5226da3784 Refactor and document add_package
36e1daf0a6 DynFlags: make listVisibleModuleNames take a UnitState
bd5810dc4e DynFlags: remove useless add_package parameter
9e715c1b84 Document getPreloadUnitsAnd
266bc3d9c3 DynFlags: refactor unwireUnit
9400aa9348 Remove preload parameter of mkUnitState
598cc1dde5 Move wiring of homeUnitInstantiations outside of mkUnitState
ae900605c4 Move dump_mod_map into initUnits
653d17bdd5 Rename Package into Unit (2)
55b4263e1a Remove ClosureUnitInfoMap
202728e529 Make ClosureUnitInfoMap uses UnitInfoMap
ed533ec217 Rename Package into Unit
f50c19b8a7 Rename listUnitInfoMap into listUnitInfo
d2109b4f10 Remove PreloadUnitId type alias
3d171cd6d5 Document and refactor `mkUnit` and `mkUnitInfoMap`
d345edfe96 Refactor WiredMap
9c5572cd29 Remove LinkerUnitId type alias
e7272d53e6 Enhance UnitId use
f6be6e432e Add allowVirtualUnits field in PackageState
8dc71f5577 Rename unsafeGetUnitInfo into unsafeLookupUnit
72d086106d Refactor homeUnit
7a02599afe Remove unused code
2517a51c0f DynFlags refactoring VIII (#17957)
cf772f19c0 Enhance Note [About units] for Backpack
1c91a7a095 Bump haddock submodule
a0ea59d641 Move Config module into GHC.Settings
566cc73f46 Move isDynLinkName into GHC.Types.Name
94e7c563ab Don't use DynFlags in showLinkerState (#17957)
cab1871ab9 Move LeadingUnderscore into Platform (#17957)
40c71c2cf3 Fix colorized error messages (#18128)
1d8f80cd64 Remove references to -package-key
de9fc995c2 Fully remove PprDebug
b3df9e780f Remove PprStyle param of logging actions
f8386c7b6a Refactor PprDebug handling
780de9e110 Use platform in Iface Binary
8bfb021958 Unit: split and rename modules
10d15f1ec4 Refactoring unit management code
ea717aa424 Factorize mungePackagePaths code
9e2c8e0e37 Refactor UnitInfo load/store from databases
69562e34fb Remove unused `emptyGenericUnitInfo`
10a2ba90aa Refactor UnitInfo
2cfc4ab971 Document backpack fields in DynFlags
747093b7c2 CmmToAsm DynFlags refactoring (#17957)
f2a98996e7 Avoid `sdocWithDynFlags` in `pprCLbl` (#17957)
ce5c2999d2 Avoid using sdocWithDynFlags (#17957)
35e43d48a9 Avoid DynFlags in Ppr code (#17957)
70be0fbcef GHC.Runtime: avoid DynFlags (#17957)
6655f93324 Use ParserFlags in GHC.Runtime.Eval (#17957)
3ca5215188 GHC.Core.Opt renaming
cc2918a040 Refactor CmmStatics
a485c3c404 Move blob handling into StgToCmm
f7597aa0c0 Testsuite: measure compiler stats for T16190
4980200255 Update Stack resolver for hadrian/build-stack
0002db1bf4 Kill wORDS_BIGENDIAN and replace it with platformByteOrder (#17957)
e54500c12d Store ComponentId details
f1a6c73d01 Merge GHC.Types.CostCentre.Init into GHC.Driver.CodeOutput
1c7c6f1afc Remove GHC.Types.Unique.Map module
0de03cd787 DynFlags refactoring III
64f2075669 Refactoring: use Platform instead of DynFlags when possible
2e82465fff Refactor CmmToAsm (disentangle DynFlags)
44fad4a925 Rename isDllName
a698997137 Use a Set to represent Ways
bc41e47123 Refactor interpreterDynamic and interpreterProfiled
8e6febcee4 Refactor GHC.Driver.Session (Ways and Flags)
18757cab04 Refactor runtime interpreter code
6880d6aa1e Disentangle DynFlags and SDoc
fa28ae95e4 Fix flag documentation (#17826)
16d643cfe6 Remove -ddump-srts flag
fb5c19122e Remove redundant case
29c701c154 Refactor package related code
bf38a20eef Call `interpretPackageEnv` from `setSessionDynFlags`
c8636da514 Fix LANG=C for readelf invocation in T14999
1ca9adbc88 Remove `parallel` check from configure.ac

# Misc

f307ed226f Revert "Remove SpecConstrAnnotation (#13681)" (#19168)
753fecd273 Display FFI labels (fix #18539)
c94c56d5a7 Export SPEC from GHC.Exts (#13681)
6dbd105421 Remove outdated note
fb544de7cd Fix parsing of PIE flags
6653e139db Fix minimal imports dump for boot files (fix #18497)
d3c2d59baf RTS: avoid overflow on 32-bit arch (#18375)
edc8d22b2e LLVM: support R9 and R10 registers
8d07c48ce3 test: fix conc038
951c1fb03d Fix unboxed-sums GC ptr-slot rubbish value (#17791)
58655b9da7 Add GHC-API logging hooks
d561c8f624 Add Cmm related hooks
40327b037f Remove outdated comment
0c114c6599 Handle large ARR_WORDS in heap census (fix #17572)
7bc3a65b46 Remove SpecConstrAnnotation (#13681)
f024b6e385 Expect T4267 to pass
dce50062e3 Rts: show errno on failure (#18033)
3445b96526 Only test T16190 with the NCG
0639dc10e2 T16190: only measure bytes_allocated
7432b327a3 Use correct option name (-opti) (fix #17314)
437265eb26 Avoid timing module map dump in initUnits
99823ed24b TH: fix Show/Eq/Ord instances for Bytes (#16457)
cd4434c8e7 Fix misleading Ptr phantom type in SerializedCompact (#15653)
8ea37b01b6 RTS: workaround a Linux kernel bug in timerfd

Stack can build packages from source with the following syntax:

extra-deps:
   git: https://github.com/me/mypackage
   commit: aaaaabbbbccccdddeeeefff123456

With Stack 1.9.* we can’t use this kind of syntax to build GHC (and the other global packages such as base or template-haskell) from source.

But as my patch has been merged, the next major release of Stack will support it! With it we can easily select the compiler repository, commit ID and Hadrian flavour with the following syntax:

compiler: ghc-git-aaaaabbbbccccdddeeeefff123456-perf
# commit ID: aaaaabbbbccccdddeeeefff123456
# Hadrian flavour: "perf"

compiler-repository: https://github.com/me/ghc
# OPTIONAL. Default compiler repository is "https://gitlab.haskell.org/ghc/ghc"

Then we can use stack build as usual.

This will be particularly useful for people developing software that depends on features only available in a fork of GHC (e.g. if it uses some GHC API not available in a released GHC).

I think it could also be used to implement custom CI for GHC. For instance, a cron job could track new commits pushed into GHC repository and could use this feature to easily try to build some packages (Stackage packages?) with the corresponding compiler to track build errors, GHC regressions, etc. To help revive something like https://perf.haskell.org/ghc/ maybe?

It could also be useful to make the use of derived compilers easier without having to provide bindists (GHCJS?).

It could also be used when the official bindist is broken (e.g. https://gitlab.haskell.org/ghc/ghc/issues/16408).

Details

Hadrian is the newer build system of GHC. An Hadrian “flavour” is a set of configuration options for a GHC build (e.g. the Integer library to use, the RTS ways, etc.). You can add or modify flavours in your GHC fork if necessary.

Stack uses hadrian/build.stack.[sh,bat] scripts to build GHC, which themselves rely on Stack and on the hadrian/stack.yaml file. It means that the bootstrap GHC compiler is installed by Stack and its version is defined in this file (e.g. at the time of writing, the resolver used in hadrian/stack.yaml means that GHC 8.6.4 is used as bootstrap compiler). Other tools such as “alex” and “happy” are managed by Stack too. This makes the build environment quite reproducible!

References

• Stack documentation:
  • https://github.com/commercialhaskell/stack/blob/master/doc/yaml_configuration.md#building-ghc-from-source-experimental
• Stack patches and ticket:
  • https://github.com/commercialhaskell/stack/issues/4567
  • https://github.com/commercialhaskell/stack/pull/4581
  • https://github.com/commercialhaskell/stack/pull/4655
• GHC patches related to this work:
  • https://gitlab.haskell.org/ghc/ghc/merge_requests/305
  • https://gitlab.haskell.org/ghc/ghc/merge_requests/651
  • https://gitlab.haskell.org/ghc/ghc/merge_requests/654
  • https://gitlab.haskell.org/ghc/ghc/merge_requests/678
  • https://gitlab.haskell.org/ghc/ghc/merge_requests/741

If like me you sometimes want to embed resources files into executable binaries generated by GHC, you may have already used file-embed package. We are not alone as it is directly used by 101 other packages on Hackage at the time of writing.

The problem is: compile time and memory usage are awful for “big” files, where “big” = a few megabytes.

Here is the result of a small benchmark I wrote showing the time needed to compile an Haskell source file that embeds a zero-filled data file of the given size with file-embed:

================================
| Benchmarking file-embed      |
================================
Warming up...
# Benchmarking size: 128b
real    0m1,394s
user    0m1,191s
sys     0m0,315s

# Benchmarking size: 3K
real    0m1,259s
user    0m1,060s
sys     0m0,269s

# Benchmarking size: 3M
real    0m7,611s
user    0m6,510s
sys     0m1,183s

# Benchmarking size: 15M
real    0m32,582s
user    0m28,553s
sys     0m3,866s

# Benchmarking size: 150M
... I had to kill the process as GHC was freezing my box (i7 930, 20GB memory)

This is a known problem, so can we do better? No suspense, the answer is “yes”. The same small benchmark also benches my shiny new buffer embedding module and gives the following results:

================================
| Benchmarking haskus-binary   |
================================
Warming up...
# Benchmarking size: 128b
real    0m1,489s
user    0m1,218s
sys     0m0,371s

# Benchmarking size: 3K
real    0m1,448s
user    0m1,179s
sys     0m0,336s

# Benchmarking size: 3M
real    0m1,477s
user    0m1,245s
sys     0m0,308s

# Benchmarking size: 15M
real    0m1,660s
user    0m1,304s
sys     0m0,409s

# Benchmarking size: 150M
real    0m3,163s
user    0m1,961s
sys     0m1,291s

# Benchmarking size: 1G
real    0m21,852s
user    0m7,123s
sys     0m7,054s

So it seems like we can. At this point I don’t suggest that you use my module as it is still a work in progress and as I don’t provide an easy way to create ByteString from the embedded file. Instead I will explain how the two approaches work and how file-embed implementation could be enhanced.

file-embed approach

When we want to embed a file with file-embed, the latter uses Template Haskell (TH) to create a literal string containing the file contents. GHC compiles literal strings into the equivalent of C arrays and we can get their addresses via TH: this is what file-embed does and it creates an expression calling unsafePackAddressLen with the appropriate address and size (see here).

Literal strings were not supposed to contain big chunks of binary data, hence it is not surprising that GHC chokes on them. #14741 tracks this issue and there is even a StaticData proposal (disclaimer: I am the author) suggesting that we add new things to GHC to improve on the situation.

My approach

Now that I’m re-reading my aforementionned StaticData proposal almost a year after writing it, the approach I use is basically what I have called “Step 2” in this document and it subsumes “Step 3”.

Suppose we want to include a file “data.bin”. We follow 3 steps:

1. Via TH, we generate an assembly file containing something like:

.section ".data"
.global "myfilesymbol"
.align 8
myfilesymbol:
   .incbin "data.bin",10,128

It basically tells the assembler to include the “data.bin” file as-is into the output binary. We specify the section where to add it (“.data” or “.rodata” for read-only files), the alignment constraint, an optional file offset/size and finally we attach a global symbol to it.

2. We include this file in the compilation chain.

We use a combination of addForeignFilePath, addTempFile and addDependentFile to add a dependency on the input file and the assembly file into the compilation chain.

There is a small issue: ForeignSrcLang doesn’t support assembly files yet. A workaround is to embed the assembly file into a C file by using GCC’s asm directive. It is suboptimal and it would be better to add direct support of assembly files to TH. I have opened a ticket (#16180) with the feature request and a merge request implementing it. I don’t know if/when it will be merged but we can use the workaround in the meantime.

3. Via TH, we generate an expression to create a Buffer (or a ByteString) from the symbol in the assembly file.

We can get access to a global symbol with a foreign import declaration such as the following one:

-- Don't forget the "&" before the symbol name
foreign import ccall "&myfilesymbol" myfile_addr :: Ptr ()

To build a ByteString, we only have to generate something like the following:

myfile :: ByteString
myfile = unsafePerformIO (unsafePackCStringLen (castPtr myfile_addr, 128))
-- the size (128 here) is known statically ---------------------------^

And that’s it.

Conclusion

Using my approach is much faster as the benchmark results show. It’s because we totally bypass GHC’s pipeline: we don’t try to apply CSE to huge binary chunks; we don’t keep these huge binary chunks into memory; they don’t end up into interface files; etc.

It should be quite portable. We would have to adapt section names (“rodata”, “data”) on non ELF architectures I guess, but otherwise it should be OK.

As a bonus, with this approach we control the data alignment (useful with vector instructions) and we can choose to make the buffer read-only or not (if one tries to write a read-only buffer, it segfaults as expected).

You can find my implementation here. It is part of a larger work-in-progress on explicit memory management documented here (chapter on buffer embedding).

If someone is up to implement this into file-embed, I guess it will make a few people happy.

Changelog

• 2018-11-08 (20h30): I’ve added the “Variant + Either” approach in reaction to a reddit discussion
• 2018-11-08 (23h00): added section about the FlowT approach

This post is a follow-up to a post Matt Parsons has just published: The Trouble with Typed Errors (discussion on Reddit).

While I totally agree with the diagnostic, I don’t fully agree with the conclusion. in particular:

In PureScript or OCaml, you can use open variant types to do this flawlessly. Haskell doesn’t have open variants, and the attempts to mock them end up quite clumsy to use in practice.

Being the author of an open Variant type (doc) in Haskell, I don’t think they are that clumsy, so let’s compare!

Variant + Either approach

First let’s rewrite the safe functions head, lookup and parse by making them return a Variant on the Left side:

{-# LANGUAGE TypeApplications #-}

import Haskus.Utils.Variant

import Prelude hiding (head,lookup)
import qualified Prelude
import Text.Read

data ParseError = ParseError deriving Show

parse :: String -> Either (V '[ParseError]) Integer
parse s = case readMaybe s of
   Just i  -> Right i
   Nothing -> Left (V ParseError)


data HeadError = ListWasEmpty deriving Show

head :: [a] -> Either (V '[HeadError]) a
head []    = Left (V ListWasEmpty)
head (x:_) = Right x

data LookupError k = KeyWasNotPresent k deriving Show

lookup :: Eq k => k -> [(k,v)] -> Either (V '[LookupError k]) v
lookup k vs = case Prelude.lookup k vs of
   Just v  -> Right v
   Nothing -> Left (V (KeyWasNotPresent k))

Now to compose these functions, we just have to write:

import Data.Bifunctor (first)

liftLeft :: LiftVariant xs ys => Either (V xs) r -> Either (V ys) r
liftLeft = first liftVariant

foo str = do
   c <- liftLeft $ head str
   r <- liftLeft $ lookup c codeMap
   liftLeft $ parse (r ++ tail str)

   where
      codeMap :: [(Char, String)]
      codeMap = [ ('x', "0x")
                , ('d', "")
                ]

1. We can fix the Variant type at the definition site:

foo :: String -> Either (V '[ParseError, LookupError Char, HeadError]) Integer

-- The order of the error types doesn't matter and we can add additional error
-- types if we want:
foo :: String -> Either (V '[Float,Int,Double,ParseError,LookupError Char,HeadError,String]) Integer

Test:

> foo "d10"
Right 10

> foo "x10"
Right 16

> foo "u10"
Left V @(LookupError Char) (KeyWasNotPresent 'u')

> foo ""
Left V @HeadError ListWasEmpty

> foo "d10X"
Left V @ParseError ParseError

2. Or if don’t give foo a type signature we can fix the Variant type when we call it:

> foo "d10" :: Either (V '[ParseError,HeadError,LookupError Char]) Integer
Right 10

-- The order of the error types still doesn't matter and we can add additional
-- error types if we want:
> foo "d10" :: Either (V '[Float,Int,Double,ParseError,LookupError Char,HeadError,String]) Integer
Right 10

3. Or we can give a generic type signature to foo:

foo :: forall es.
   ('[HeadError,ParseError,LookupError Char] :<< es
   ) => String -> Either (V es) Integer

It allows us to use TypeApplications to pass the list of error types:

> foo @'[ParseError,HeadError,LookupError Char] "d10"
Right 10

> foo @'[HeadError,LookupError Char,ParseError] "d10X"
Left V @ParseError ParseError

ExceptT-like approach

Just like ExceptT wraps Either, we can use a FlowT newtype to wraps a Variant. Compared to the approach above (“Either + Variant”), we avoid the Either indirection: the first value of the Variant is considered as the Right value and the other ones as the error values.

The example above becomes:

{-# LANGUAGE TypeApplications #-}
{-# LANGUAGE TypeOperators #-}

import Haskus.Utils.Variant.Flow

import Prelude hiding (head,lookup)
import qualified Prelude
import Text.Read

data ParseError = ParseError deriving Show

parse :: String -> Flow '[ParseError] Integer
parse s = case readMaybe s of
   Just i  -> pure i
   Nothing -> throwE ParseError


data HeadError = ListWasEmpty deriving Show

head :: [a] -> Flow '[HeadError] a
head []    = throwE ListWasEmpty
head (x:_) = pure x

data LookupError k = KeyWasNotPresent k deriving Show

lookup :: Eq k => k -> [(k,v)] -> Flow '[LookupError k] v
lookup k vs = case Prelude.lookup k vs of
   Just v  -> pure v
   Nothing -> throwE (KeyWasNotPresent k)


foo :: String -> Flow '[ParseError, LookupError Char, HeadError] Integer
-- foo :: forall es.
--    ('[HeadError,ParseError,LookupError Char] :<< es
--    ) => String -> Flow es Integer
foo str = do
   c <- liftFlowT $ head str
   r <- liftFlowT $ lookup c codeMap
   liftFlowT $ parse (r ++ tail str)

   where
      codeMap :: [(Char, String)]
      codeMap = [ ('x', "0x")
                , ('d', "")
                ]

Test:

> runFlow (foo "d10")
V @Integer 10

> runFlow (foo "x10")
V @Integer 16

> runFlow (foo "u10")
V @(LookupError Char) (KeyWasNotPresent 'u')

> runFlow (foo "")
V @HeadError ListWasEmpty

> runFlow (foo "d10X")
V @ParseError ParseError

> foo "" `catchE` (\ListWasEmpty -> success 42) :: Flow '[ParseError,LookupError Char] Integer
FlowT (Identity V @Integer 42)

(FlowT is available from release 2.4 of haskus-utils-variant package).

Full Variant approach

Finally, there is another approach that directly uses Variant with RebindableSyntax extension.

First let’s rewrite the safe functions head, lookup and parse by making them return a Variant:

{-# LANGUAGE TypeApplications #-}

import Haskus.Utils.Variant

import Prelude hiding (head,lookup)
import qualified Prelude
import Text.Read

data ParseError = ParseError deriving Show

parse :: String -> V '[Integer,ParseError]
parse s = case readMaybe s of
   Just i  -> V @Integer i    -- we use the `V` pattern to index the
   Nothing -> V ParseError    -- Variant by type


data HeadError = ListWasEmpty deriving Show

head :: [a] -> V '[a,HeadError]
head []    = toVariantAt @1 ListWasEmpty  -- we can't index the Variant by
head (x:_) = toVariantAt @0 x             -- type because `a` is ambiguous,
                                          -- so we do it by index explicitly

data LookupError k = KeyWasNotPresent k deriving Show

lookup :: Eq k => k -> [(k,v)] -> V '[v,LookupError k]
lookup k vs = case Prelude.lookup k vs of
   Just v  -> toVariantAt @0 v            -- ditto
   Nothing -> toVariantAt @1 (KeyWasNotPresent k)

Now we could use Variant’s primitives to compose these functions (again see the documentation). But instead I will show how we can use them implicitly with do-notation thanks to the RebindableSyntax extension:

{-# LANGUAGE RebindableSyntax #-}

import Haskus.Utils.Variant.Syntax

import Prelude hiding ((>>=),(>>),return)

foo :: String -> V '[Integer, ParseError, LookupError Char, HeadError]
foo str = do
   c <- head str
   r <- lookup c codeMap
   parse (r ++ tail str)

   where
      codeMap :: [(Char, String)]
      codeMap = [ ('x', "0x")
                , ('d', "")
                ]

Test:

> foo "d10"
V @Integer 10

> foo "x10"
V @Integer 16

> foo "u10"
V @(LookupError Char) (KeyWasNotPresent 'u')

> foo ""
V @HeadError ListWasEmpty

> foo "d10X"
V @ParseError ParseError

Conclusion

It doesn’t look that clumsy to me! ;-)

A short post to announce that I have just released version 2.0.1 of my haskus-utils-variant package (containing Variant and EADT).

The documentation has also been greatly enhanced thanks to the feedback I’ve received:

• Variant: https://docs.haskus.org/variant.html
• EADT: https://docs.haskus.org/eadt.html

As a short introduction, a Variant is a generic sum type (i.e. like Either but supporting any number of value types instead of just 2):

w,x,y,z :: V '[String,Int,Float,Maybe String]
w = V "test"
x = V @Int 10
y = V @Float 5.0
z = V (Just "Hello World")

Similarly, an EADT is an extensible algebraic data type (ADT). Compared to Variant, it supports recursively defined ADT. For instance a list EADT can be defined with:

{-# LANGUAGE TemplateHaskell #-}
{-# LANGUAGE DeriveFunctor #-}
{-# LANGUAGE PatternSynonyms #-}
{-# LANGUAGE TypeFamilies #-}

data ConsF a l = ConsF a l deriving (Functor)
data NilF    l = NilF      deriving (Functor)

eadtPattern 'ConsF "Cons"
eadtPattern 'NilF  "Nil"

type List a = EADT '[ConsF a, NilF]

x :: List String
x = Cons "Hello" (Cons "World" Nil)

Of course the whole point of EADT is not to redefine lists in a weird way but to help with codes where we would like to add or remove ADT constructors (e.g. different abstract syntax trees (AST) in a compiler) or to support generic recursive object hierarchies (e.g. extensible GUI widget hierarchies).

Any feedback appreciated!

Introduction

The expression problem (1998)

In 1998, Philip Wadler defined the Expression Problem as follow:

The Expression Problem is a new name for an old problem. The goal is to define a datatype by cases, where one can add new cases to the datatype and new functions over the datatype, without recompiling existing code, and while retaining static type safety

In basic Haskell it is straightforward to add new functions over a data type. Suppose we have the following arithmetic expression data type:

data Expr = Val Int | Add Expr Expr

We can independently add an evaluator function, potentially in another module:

eval :: Expr -> Int
eval (Val x)   =  x
eval (Add x y) = eval x + eval y

However if we want to add a new case to the data type (say support for multiplication), we have to modify both the data type definition and the functions using it:

data Expr = .... | Mul Expr Expr

eval :: Expr -> Int
....
eval (Mul x y) = eval x * eval y

Data types à la carte (2008)

Ten years later (in 2008), Wouter Swierstra described a technique to handle this in his well-known Data types à la carte paper. The first idea is to define data constructors independently and to use a type parameter to leave open the data type they are part of.

-- Independent data constructors. Parameter `e` represents the data type they
-- will be part of. It is required even if it is not used in the right hand
-- side.
data Val e = Val Int
data Add e = Add e e

Defining a new independent constructor is easy:

data Mul e = Mul e e

The second idea is to use a combining data type:

data (f :+: g) e = Inl (f e) | Inr (g e)

It is similar to Either except that it passes the same additional type parameter to both f and g type constructors. It can be used to compose independent data constructors without creating a new data type:

type ExprF = Val :+: Add

ExprF has kind Type -> Type and its type parameter is used as the e parameter of the independent data constructors. We can set it to arbitrary types such as Int to build valid values:

y = Inr (Add 5 8) :: ExprF Int

However the main use of this parameter should be to indicate the type of the expression data type we want to build, say Expr. Hence we would like to write something like this:

type Expr = ExprF Expr

 >error:
 Cycle in type synonym declarations:
   <interactive>:12:1-22: type Expr = ExprF Expr

Oops, we can’t build this cyclic (infinite) type. This leads us to the third idea: use another data type to handle the recursive nature of the expression type:

data Expr = Expr (ExprF Expr)

We can abstract on it to use the same data type for different expression types:

-- `Fix` type as defined in Data.Functor.Foldable for instance
newtype Fix f = Fix (f (Fix f))

type Expr = Fix ExprF

Summary

We use 3 different kinds of data types:

1. Constructor data types: Val, Add, Mul…
2. Combinator data type: :+:
3. Recursivity handling data type: Fix

By using these different data types we have untangled the construction of ADTs (algebraic data types) and we can freely add new constructor data types and mix them into different algebraic data types.

Operations on these algebraic data types can be defined independently by using type-classes and recursion schemes.

EADT - Extensible ADT (2018)

In this post I’d like to slightly revisit and enhance the approach by using modern Haskell and in particular another combinator data type and pattern synonyms.

A different combinator data type

Variant (xs :: [*]) is a sum type that contains a value whose type can be any type in the xs list (see this previous post). We can use it to define the following VariantF type:

newtype VariantF (xs :: [* -> *]) e = VariantF (Variant (ApplyAll e xs))

-- ApplyAll e '[f,g,h] ==> '[f e, g e, h e]

instance Functor (VariantF xs) where ....

We will use this type as a combinator data type. Similarly to the :+: combinator data type presented in the introduction, VariantF passes its e parameter to all of its “member” types and has an instance of the Functor class.

Now instead of writing f :+: g :+: h :+: i to combine constructor data types to form an ADT we can write VariantF '[f,g,h,i].

First benefit: just like using Variant is more efficient – O(1) memory usage and (de)construction – than using a nest of Either, using VariantF is more efficient than using a nest of :+:.

A generic EADT type

Let’s define an EADT type synonym as follow:

type EADT xs = Fix (VariantF xs)

And a pattern synonym to easily build EADT values:

pattern VF :: forall e f cs.
   ( e ~ EADT cs  -- allow easy use of TypeApplication to set the EADT type
   , f :<: cs     -- constraint synonym ensuring `f` is in `cs`
   ) => f (EADT cs) -> EADT cs
pattern VF x = Fix (VariantF (V' x))   -- `V'` match a variant value (without
                                       -- checking the membership: we already
                                       -- do it with :<:)

Second benefit: with modern Haskell we can define bidirectional pattern synonyms that make creation and matching of EADT values much more nicer.

Examples using EADT

Defining an EADT and building some values

Let’s define some constructors:

{-# LANGUAGE DeriveFunctor #-}

data ValF e = ValF Int deriving (Functor)
data AddF e = AddF e e deriving (Functor)

And an algebraic data type:

type AddValADT = EADT '[ValF,AddF]

We can build EADT values in the following (equivalent) ways:

x :: AddValADT
x = VF (ValF 10)

y = VF @AddValADT (ValF 10)

z = VF (ValF 10 :: ValF AddValADT)

But it’s nicer to define some additional pattern synonyms:

pattern Val :: ValF :<: xs => Int -> EADT xs
pattern Val a = VF (ValF a)

pattern Add :: AddF :<: xs => EADT xs -> EADT xs -> EADT xs
pattern Add a b = VF (AddF a b)

x :: AddValADT
x = Val 10

xx = Add (Val 10) (Val 5) :: AddValADT

We can pattern-match EADT values with VF or with pattern synonyms too:

showAddValADT :: AddValADT -> String
showAddValADT e = case e of
   -- match with VF
   VF (ValF u)  -> show u

   -- match with additional pattern synonym
   Add u v      -> "(" ++ showAddValADT u ++ " + " ++ showAddValADT v ++ ")"

   _            -> error "showAddValADT: unhandled AddValADT constructor"

> showAddValADT (Add (Val 10) (Val 3))
"(10 + 3)"

Note that the wildcard match is necessary, otherwise the pattern-match is reported as non exhaustive by GHC (warning). This is one shortcoming due to the use of the Variant type.

Anyway it isn’t really an issue in this case because showAddValADT shouldn’t be written like this to be extensible.

Extending an EADT and converting values from one EADT to another

Suppose we add a constructor:

data MulF e = MulF e e deriving (Functor)

We can define a new EADT:

type MulAddValADT = EADT '[ValF,AddF,MulF]

We only need to change the type to declare values of the new EADT:

x' :: MulAddValADT
x' = Val 10

We can explicitly add constructors to an EADT:

> x' = Val 10 :: EADT '[ValF,AddF]

> :t appendEADT @'[MulF] x'
appendEADT @'[MulF] x' :: EADT '[ValF, AddF, MulF]

We can also lift values from one EADT to the other as follow:

xx' :: MulAddValADT
xx' = liftEADT (Add (Val 10) (Val 5) :: AddValADT)

Thanks to the type-checking, the opposite direction is not allowed:

> liftEADT xx' :: AddValADT

<interactive>:83:1: error:
    • '[MulF]
      is not a subset of
      '[ValF, AddF]
    • In the expression: liftEADT xx' :: AddValADT
      In an equation for ‘it’: it = liftEADT xx' :: AddValADT

Defining operations on EADTs

Extensible operations on EADTS (i.e. that need to be extended when we add new constructors) are defined with type-classes.

Type classes and recursion schemes

We can use recursion schemes, for example:

import Data.Functor.Foldable

class ShowEADT (f :: * -> *) where
   showEADT' :: f String -> String

-- boilerplate instances for the combinator data type

instance ShowEADT (VariantF '[]) where
   showEADT' = undefined

instance (ShowEADT x, ShowEADT (VariantF xs))  => ShowEADT (VariantF (x ': xs)) where
   showEADT' v = case popVariantFHead v of
      Right u -> showEADT' u
      Left  w -> showEADT' w


-- instances for constructors data types

instance ShowEADT ValF where
   showEADT' (ValF i) = show i

instance ShowEADT AddF where
   showEADT' (AddF u v) = "(" ++ u ++ " + " ++ v ++ ")" -- no recursive call

instance ShowEADT MulF where
   showEADT' (MulF u v) = "(" ++ u ++ " * " ++ v ++ ")" -- no recursive call


-- `cata` (for catamorphism) handles the recursion/fold

> cata showEADT' (Add (Val 2) (Add (Val 10) (Val 3)) :: AddValADT)
"(2 + (10 + 3))"

> cata showEADT' (Add (Val 2) (Add (Val 10) (Val 3)) :: MulAddValADT)
"(2 + (10 + 3))"

> cata showEADT' (Mul (Val 2) (Add (Val 10) (Val 3)) :: MulAddValADT)
"(2 * (10 + 3))"


showEADT :: (ShowEADT (Base t), Recursive t) => t -> String -- type inferred by GHC
showEADT = cata showEADT'

> showEADT (Mul (Val 2) (Add (Val 10) (Val 3)) :: MulAddValADT)
"(2 * (10 + 3))"

Type classes and explicit recursion

We can also use explicit recursion, for example:

class Eval e where
   -- evaluate an expression
   eval :: e -> Int

-- same boilerplate

instance Eval (VariantF '[] e) where
   eval = undefined

instance (Eval (x e), Eval (VariantF xs e))  => Eval (VariantF (x ': xs) e) where
   eval v = case popVariantFHead v of
      Right u -> eval u
      Left  w -> eval w

-- evaluating an EADT value

type EvalAll xs = Eval (VariantF xs (EADT xs))

evalEADT :: EvalAll xs => EADT xs -> Int
evalEADT = eval . unfix

-- instances for constructors

instance Eval (ValF e) where
   eval (ValF i) = i

instance EvalAll xs => Eval (AddF (EADT xs)) where
   eval (AddF u v) = evalEADT u + evalEADT v -- explicit recursion

instance EvalAll xs => Eval (MulF (EADT xs)) where
   eval (MulF u v) = evalEADT u * evalEADT v -- explicit recursion


> evalEADT (Add (Val 2) (Add (Val 10) (Val 3)) :: AddValADT)
15
> evalEADT (Add (Val 2) (Add (Val 10) (Val 3)) :: MulAddValADT)
15
> evalEADT (Mul (Val 2) (Add (Val 10) (Val 3)) :: MulAddValADT)
26

Transformation example

Let’s rewrite Swiestra’s example to apply the distributivity law. The operation is not extensible, hence we don’t have to use type classes to define it. It works on any EADT as long as the constraints are fulfilled: supporting Add and Mul constructors.

import Control.Arrow

-- distribute multiplication over addition if it matches
distr :: (AddF :<: f, MulF :<: f) => EADT f -> Maybe (EADT f)
distr (Mul a (Add c d)) = Just (Add (Mul a c) (Mul a d))
distr _                 = Nothing

-- bottom up traversal that performs an additional bottom up traversal in
-- the transformed sub-tree when a transformation occurs. 
bottomUpFixed :: Functor f => (Fix f -> Maybe (Fix f)) -> Fix f -> Fix f
bottomUpFixed f = unfix >>> fmap (bottomUpFixed f) >>> Fix >>> f'
   where
      f' u = case f u of
         Nothing -> u
         Just v  -> bottomUpFixed f v

-- example
expr :: MulAddValADT
expr = Mul (Val 2) (Add (Val 10) (Mul (Val 3) (Add (Val 5) (Val 20))))

> showEADT expr
"(2 * (10 + (3 * (5 + 20))))"

> showEADT (bottomUpFixed distr expr)
"((2 * 10) + ((2 * (3 * 5)) + (2 * (3 * 20))))"

Constructor removal example (~desugaring)

Let’s show how to transform each multiplication into an addition while removing the MulF constructor from the EADT in a generic type-safe manner:

class MulToAdd (f :: * -> *) ys where
   mulToAdd :: f (EADT ys) -> EADT ys

-- boilerplate

instance MulToAdd (VariantF '[]) ys where
   mulToAdd = undefined

instance (MulToAdd x ys, MulToAdd (VariantF xs) ys)  => MulToAdd (VariantF (x ': xs)) ys where
   mulToAdd v = case popVariantFHead v of
      Right u -> mulToAdd u
      Left  w -> mulToAdd w

-- constructor instances

instance{-# OVERLAPPING #-} AddF :<: ys => MulToAdd MulF ys where
   mulToAdd (MulF u v) = Add u v -- Mul becomes Add
                                 -- (it makes no sense but it's just an example)

instance {-# OVERLAPPABLE #-} f :<: ys => MulToAdd f ys where
   mulToAdd = VF  -- the other constructors are kept as-is

-- example
expr :: MulAddValADT
expr = Mul (Val 2) (Add (Val 10) (Mul (Val 3) (Add (Val 5) (Val 20))))

> showEADT (cata mulToAdd expr :: AddValADT)
"(2 + (10 + (3 + (5 + 20))))"

> showEADT (cata mulToAdd (bottomUpFixed distr expr) :: AddValADT)
"((2 + 10) + ((2 + (3 + 5)) + (2 + (3 + 20))))"

This kind of transformation can be used to transform a big language into a smaller one (e.g. Haskell into Core).

Annotation examples (parameter example)

Let’s add a constructor to support annotations in the AST. We let the annotation type parametric.

-- | Annotate `e` with a value having type `a`
data AnnotF a e = AnnotF a e deriving (Functor)

pattern Annot :: AnnotF a :<: xs => a -> EADT xs -> EADT xs
pattern Annot a e = VF (AnnotF a e)

instance Show a => ShowEADT (AnnotF a) where
   showEADT' (AnnotF a e) = "{" ++ show a ++ "} " ++ e

We can easily define an EADT supporting parametric annotations, for instance:

type AnnotExp a = EADT '[ValF,AddF,MulF,AnnotF a]

Now we can build an example of an annotated expression where the annotation type is String:

annotExpr :: AnnotExp String
annotExpr = Mul (Val 8) (Annot "An addition" (Add (Val 10) (Val 5)))

> showEADT annotExpr 
"(8 * {\"An addition\"} (10 + 5))"

We can easily prune annotations from the expression similarly to what we did above to transform multiplications into additions:

class RemoveAnnot (f :: * -> *) ys where
   removeAnnot' :: f (EADT ys) -> EADT ys

instance RemoveAnnot (VariantF '[]) ys where
   removeAnnot' = undefined

instance
   ( RemoveAnnot x ys
   , RemoveAnnot (VariantF xs) ys
   )  => RemoveAnnot (VariantF (x ': xs)) ys
   where
   removeAnnot' v = case popVariantFHead v of
      Right u -> removeAnnot' u
      Left  w -> removeAnnot' w

instance{-# OVERLAPPING #-} RemoveAnnot (AnnotF a) ys where
   removeAnnot' (AnnotF _a e) = e

instance {-# OVERLAPPABLE #-} f :<: ys => RemoveAnnot f ys where
   removeAnnot' = VF

> showEADT (cata removeAnnot' annotExpr :: MulAddValADT)
"(8 * (10 + 5))"

As it is a bit sad to have to specify the type of the resulting EADT without the annotations, we can define the following helper function:

-- remove `AnnotF a` (for all `a`) in `xs`
type family FilterAnnot xs where
   FilterAnnot '[]              = '[]
   FilterAnnot (AnnotF a ': xs) = FilterAnnot xs
   FilterAnnot (x ': xs)        = x ': FilterAnnot xs

removeAnnot ::
   ( Functor (VariantF xs)
   , RemoveAnnot (VariantF xs) (FilterAnnot xs)
   ) => EADT xs -> EADT (FilterAnnot xs)
removeAnnot = cata removeAnnot'

> showEADT (removeAnnot annotExpr)
"(8 * (10 + 5))"

Type-checking example (paramorphism example)

First let’s add support for Float values:

data FloatValF e = FloatValF Float deriving (Functor)

pattern FloatVal :: FloatValF :<: xs => Float -> EADT xs
pattern FloatVal a = VF (FloatValF a)

instance ShowEADT FloatValF where
   showEADT' (FloatValF i) = show i

Now we can write invalid expressions mixing both Int and Float values:

> showEADT (Add (Val 10) (FloatVal 5) :: EADT '[ValF,FloatValF,AddF])
"(10 + 5.0)"

Let’s write a type-checker to detect these cases. For a given expression, our type-checker will either return its valid type (Int or Float) or an erroneous type with an explanation message:

data Typ
   = TInt
   | TFloat
   | TError String   -- the type of an invalid expression with some explanation
   deriving (Show,Eq)

class TypeCheck (f :: * -> *) where
   typeCheck' :: f Typ -> Typ

instance TypeCheck (VariantF '[]) where
   typeCheck' = undefined

instance (TypeCheck x, TypeCheck (VariantF xs))  => TypeCheck (VariantF (x ': xs)) where
   typeCheck' v = case popVariantFHead v of
      Right u -> typeCheck' u
      Left  w -> typeCheck' w

instance TypeCheck ValF where
   typeCheck' _ = TInt

instance TypeCheck FloatValF where
   typeCheck' _ = TFloat

instance TypeCheck AddF where
   typeCheck' (AddF t1 t2)
      | t1 == t2       = t1
      | TError _ <- t1 = t1 -- propagate errors
      | TError _ <- t2 = t2
      | otherwise      = TError $ "can't match " ++ show t1 ++ " with " ++ show t2

Let’s try it:

> cata typeCheck' (Add (Val 10) (Val 5) :: EADT '[ValF,FloatValF,AddF])
TInt

> cata typeCheck' (Add (Val 10) (FloatVal 5) :: EADT '[ValF,FloatValF,AddF])
TError "can't match TInt with TFloat"

The error message is not very useful, let’s write a better type-checker that indicates which expressions are at fault:

class TypeCheck2 (f :: * -> *) ys where
   typeCheck2' :: f (EADT ys, Typ) -> Typ

instance TypeCheck2 (VariantF '[]) ys where
   typeCheck2' = undefined

instance (TypeCheck2 x ys, TypeCheck2 (VariantF xs) ys)  => TypeCheck2 (VariantF (x ': xs)) ys where
   typeCheck2' v = case popVariantFHead v of
      Right u -> typeCheck2' u
      Left  w -> typeCheck2' w

instance TypeCheck2 ValF ys where
   typeCheck2' _ = TInt

instance TypeCheck2 FloatValF ys where
   typeCheck2' _ = TFloat

instance (AddF :<: ys, ShowEADT (VariantF ys), Functor (VariantF ys)) => TypeCheck2 AddF ys where
   typeCheck2' (AddF (u,t1) (v,t2))
      | t1 == t2       = t1
      | TError _ <- t1 = t1 -- propagate errors
      | TError _ <- t2 = t2
      | otherwise = TError $ "can't add `" ++ showEADT u ++ "` whose type is " ++ show t1 ++
                             " with `" ++ showEADT v ++ "` whose type is " ++ show t2

This time instead of using cata we will use another recursion scheme called para (for paramorphism) that also passes expression values (u and v in AddF instance):

> para typeCheck2' (Add (Val 10) (Val 5) :: EADT '[ValF,FloatValF,AddF])
TInt

> para typeCheck2' (Add (Val 10) (FloatVal 5) :: EADT '[ValF,FloatValF,AddF])
TError "can't add `10` whose type is TInt with `5.0` whose type is TFloat"

Let’s check that we can easily extend our expression language and our type-checker without rewriting anything by adding support for type ascriptions (i.e., types specified by the programmer):

data TypedF e = TypedF Typ e deriving (Functor)

pattern Typed :: TypedF :<: xs => Typ -> EADT xs -> EADT xs
pattern Typed t e = VF (TypedF t e)

instance ShowEADT TypedF where
   showEADT' (TypedF t e) = "(" ++ e ++ " :: " ++ show t ++ ")"

instance (Functor (VariantF ys), ShowEADT (VariantF ys)) => TypeCheck2 TypedF ys where
   typeCheck2' (TypedF t (e,te))
      | t == te   = t
      | TError _ <- t  = t
      | TError _ <- te = te
      | otherwise = TError $ "wrong type ascription " ++ show t ++
                             " for expression `" ++ showEADT e ++
                             "` with inferred type " ++ show te

Now let’s check that asserting some types is correctly supported:

> type E = EADT '[ValF,FloatValF,AddF,TypedF]

> para typeCheck2' (Add (Val 10) (Typed TInt $ Val 5) :: E)
TInt

> para typeCheck2' (Add (Val 10) (Typed TFloat $ Val 5) :: E)
TError "wrong type ascription TFloat for expression `5` with inferred type TInt"

> para typeCheck2' (Typed TInt $ Add (Val 10) (Val 5) :: E)
TInt

> para typeCheck2' (Typed TFloat $ Add (Val 10) (Val 5) :: E)
TError "wrong type ascription TFloat for expression `(10 + 5)` with inferred type TInt"

Finally let’s write another type-checker that adds inferred types as type-ascriptions to every node of the expression.

class TypeCheck3 (f :: * -> *) ys where
   typeCheck3' :: f (TypedF (EADT ys)) -> TypedF (EADT ys)

instance TypeCheck3 (VariantF '[]) ys where
   typeCheck3' = undefined

instance (TypeCheck3 x ys, TypeCheck3 (VariantF xs) ys)  => TypeCheck3 (VariantF (x ': xs)) ys where
   typeCheck3' v = case popVariantFHead v of
      Right u -> typeCheck3' u
      Left  w -> typeCheck3' w

instance (TypedF :<: ys, ValF :<: ys) => TypeCheck3 ValF ys where
   typeCheck3' (ValF i) = TypedF TInt (Val i)

instance (TypedF :<: ys, FloatValF :<: ys) => TypeCheck3 FloatValF ys where
   typeCheck3' (FloatValF f) = TypedF TFloat (FloatVal f)

instance
   ( TypedF :<: ys
   , AddF :<: ys
   , ShowEADT (VariantF ys)
   , Functor (VariantF ys)
   ) => TypeCheck3 AddF ys
   where
   typeCheck3' (AddF u@(TypedF t1 _) v@(TypedF t2 _))
      | t1 == t2       = TypedF t1 addVal
      | TError _ <- t1 = TypedF (TError "") addVal -- don't propagate errors
      | TError _ <- t2 = TypedF (TError "") addVal
      | otherwise = TypedF (TError $ "can't add " ++ show t1 ++ " with " ++ show t2) addVal
      where
         addVal = Add (VF u) (VF v)

-- similar for MulF...

instance
   ( TypedF :<: ys
   , Functor (VariantF ys)
   , ShowEADT (VariantF ys)
   ) => TypeCheck3 TypedF ys
   where
   typeCheck3' (TypedF t u@(TypedF te _))
      | t == te        = u                         -- Remove redundant ascription
      | TError _ <- t  = TypedF t (VF u)           -- keep erroring ascription as-is
      | TError _ <- te = TypedF (TError "") (VF u) -- don't propagate errors
      | otherwise      = TypedF (TError $ "wrong type ascription" ++
                                          " (inferred type: " ++ show te ++ ")") (Typed t (VF u))


typeCheck3 :: forall xs.
   ( TypeCheck3 (VariantF xs) xs
   , Functor (VariantF xs)
   , TypedF :<: xs
   ) => EADT xs -> EADT xs
typeCheck3 e = VF (cata typeCheck3' e :: TypedF (EADT xs))

Let’s try it:

> type E = EADT '[ValF,FloatValF,AddF,MulF,TypedF]

> showEADT (typeCheck3 (Add (Val 10) (Val 5) :: E))
"(((10 :: TInt) + (5 :: TInt)) :: TInt)"

> showEADT (typeCheck3 (Add (Val 10) (FloatVal 5) :: E))
"(((10 :: TInt) + (5.0 :: TFloat)) :: TError \"can't add TInt with TFloat\")"

> showEADT (typeCheck3 (Add (Add (Mul (Val 2) (Val 3)) (FloatVal 10)) (Val 5)) :: E)
"(((((((2 :: TInt) * (3 :: TInt)) :: TInt) + (10.0 :: TFloat))
   :: TError \"can't add TInt with TFloat\") + (5 :: TInt)) :: TError \"\")"

It’s a bit hard to read, let’s add an helper:

import Data.Tree

class DisplayEADT (f :: * -> *) where
   displayEADT' :: f (Tree String) -> Tree String

instance DisplayEADT (VariantF '[]) where
   displayEADT' = undefined

instance (DisplayEADT x, DisplayEADT (VariantF xs))  => DisplayEADT (VariantF (x ': xs)) where
   displayEADT' v = case popVariantFHead v of
      Right u -> displayEADT' u
      Left  w -> displayEADT' w

instance DisplayEADT ValF where
   displayEADT' (ValF i) = Node (show i) []

instance DisplayEADT FloatValF where
   displayEADT' (FloatValF i) = Node (show i) []

instance DisplayEADT AddF where
   displayEADT' (AddF u v) = Node "(+)" [u,v]

instance DisplayEADT MulF where
   displayEADT' (MulF u v) = Node "(*)" [u,v]

instance DisplayEADT TypedF where
   displayEADT' (TypedF t (Node l ts)) = Node (l ++ " :: " ++ show t) ts

drawEADT :: (Functor (VariantF xs), DisplayEADT (VariantF xs)) => EADT xs -> IO ()
drawEADT = putStr . drawTree . cata displayEADT'

And try it:

> drawEADT (typeCheck3 (Add (Add (Mul (Val 2) (Val 3)) (FloatVal 10)) (Val 5)) :: E)

(+) :: TError ""
|
+- (+) :: TError "can't add TInt with TFloat"
|  |
|  +- (*) :: TInt
|  |  |
|  |  +- 2 :: TInt
|  |  |
|  |  `- 3 :: TInt
|  |
|  `- 10.0 :: TFloat
|
`- 5 :: TInt

Composition example

Finally, we can write the following one-line example that strips annotations, adds support for type-ascriptions, performs type-checking and displays the resulting EADT:

annotExpr :: EADT '[ValF,AddF,MulF,AnnotF String]
annotExpr = Mul (Val 8) (Annot "An addition" (Add (Val 10) (Val 5)))

> drawEADT (typeCheck3 (appendEADT @'[TypedF] (removeAnnot annotExpr)))

(*) :: TInt
|
+- 8 :: TInt
|
`- (+) :: TInt
   |
   +- 10 :: TInt
   |
   `- 5 :: TInt

The compiler ensures that the EADT is valid at each phase. For instance if we forget to remove annotations, type-checking and drawing phases will complain at compile time:

> drawEADT (typeCheck3 (appendEADT @'[TypedF] annotExpr))

<interactive>:367:1: error:
    • No instance for (DisplayEADT (AnnotF String))
        arising from a use of ‘drawEADT’
    • In the expression:
        drawEADT (typeCheck3 (appendEADT @'[TypedF] annotExpr))
      In an equation for ‘it’:
          it = drawEADT (typeCheck3 (appendEADT @'[TypedF] annotExpr))

<interactive>:367:11: error:
    • No instance for (TypeCheck3
                         (AnnotF String) '[ValF, AddF, MulF, AnnotF String, TypedF])
        arising from a use of ‘typeCheck3’
    • In the first argument of ‘drawEADT’, namely
        ‘(typeCheck3 (appendEADT @'[TypedF] annotExpr))’
      In the expression:
        drawEADT (typeCheck3 (appendEADT @'[TypedF] annotExpr))
      In an equation for ‘it’:
          it = drawEADT (typeCheck3 (appendEADT @'[TypedF] annotExpr))

This way we can easily encode phase invariants in the types.

Remarks

This post has been motivated by a discussion on Reddit about AST transformations: each phase of a compiler transforms an AST into another and there is a tension between using a different AST data type per phase (best approach if we consider phases independently) or a generic AST data type that can be used by every phase (best approach to easily compose/interleave phases).

The usual answer (as used by GHC) is taken from the “Trees that grow” paper and consists in using an ADT as a generic AST and to parameterize it with type families. I don’t really like it and I think that using EADTs instead could probably avoid the use of AST specific type-families and hence could make the AST more generic (easier to add phases, etc.).

I have used recursion schemes in this post without really introducing them. Patrick Thomson wrote a very good series of posts about them that I highly recommend.

Please let me know if you do some fun or interesting things with EADTs!

Sources can be found here: https://github.com/haskus/haskus-utils in Haskus.Utils.VariantF.

ADDENDUM

Following the discussion on Reddit about this post, I have implemented the speculated “project” function from the Compositional data types paper (Future Work section) with the proposed type. I have called it popEADT and it allows us to perform complete case analysis (with its friends popVariantF and variantFToValue), e.g.:

showMulAddValADT :: MulAddValADT -> String
showMulAddValADT e = case popEADT e of
   Right (ValF u) -> show u
   Left w         -> case popVariantF w of
      Right (AddF u v) ->  "(" ++ showMulAddValADT u ++ " + " ++ showMulAddValADT v ++ ")"
      Left z           -> case variantFToValue z of
         MulF u v ->  "(" ++ showMulAddValADT u ++ " * " ++ showMulAddValADT v ++ ")"

> v = Mul (Add (Val 2) (Val 3)) (Val 5) :: MulAddValADT 
> showMulAddValADT v
"((2 + 3) * 5)"

Notice that contrary to the previous showAddValADT example in the post, there is no need for any extra wildcard match this time because GHC detects that the pattern-matching is complete.

This post is about managing collections (e.g., lists) of heterogeneous data types in Haskell.

Summary: in my opinion, using the data type I’ve called Variant to handle heterogeneous collections is currently the best alternative. It is type-safe, efficient (both storage-wise and performance-wise) and easy to use.

The problem

Suppose we have several algebraic data types for different geometric shapes:

data Circle = Circle
         { circleRadius :: Float
         , circleCenter :: (Float,Float)
         }
         deriving (Show)

data Square = Square
         { squareTopLeft :: (Float,Float)
         , squareWidth   :: Float
         }
         deriving (Show)

and that we know how to resize them by a factor:

class Resizable a where
   resize :: Float -> a -> a

instance Resizable Circle where
   resize f (Circle r c) = Circle (f*r) c

instance Resizable Square where
   resize f (Square tl w) = Square tl (f*w)

We can write a function to resize a list of shapes:

resizeAll :: Resizable a => Float -> [a] -> [a]
resizeAll f = fmap (resize f)

Let’s test it on a collection of circles:

> resizeAll 2 [Circle 5 (1,2), Circle 4 (2,3)]

[Circle {circleRadius = 10.0, circleCenter = (1.0,2.0)}
,Circle {circleRadius = 8.0, circleCenter = (2.0,3.0)}]

Now on a collection of squares:

> resizeAll 2 [Square (1,2) 5, Square (2,3) 4]

[Square {squareTopLeft = (1.0,2.0), squareWidth = 10.0}
,Square {squareTopLeft = (2.0,3.0), squareWidth = 8.0}]

And finally on a collection mixing circles and squares:

> resizeAll 2 [Square (1,2) 5, Circle 4 (2,3)]

<interactive>:10:30: error:
    • Couldn't match expected type ‘Square’ with actual type ‘Circle’
    • In the expression: Circle 4 (2, 3)
      In the second argument of ‘resizeAll’, namely
        ‘[Square (1, 2) 5, Circle 4 (2, 3)]’
      In the expression: resizeAll 2 [Square (1, 2) 5, Circle 4 (2, 3)]

Argh! We forgot that lists can only contain elements having the same type! We would like an heterogeneous collection that can contain both squares and circles (and maybe more shapes)!

Heterogeneous Collections

We can use the Variant data type (or its alias V) and the pattern synonym V to wrap our values:

import Haskus.Utils.Variant

-- a collection of circles and squares
shapes :: [V '[Circle,Square]]
shapes = [ V $ Square (1,2) 5
         , V $ Circle 4 (2,3)
         , V $ Circle 1 (5,7)
         ]


data Triangle = Triangle deriving (Show)

-- a collection of circles, squares and triangles
shapes2 :: [V '[Circle,Square,Triangle]]
shapes2 = [ V $ Square (1,2) 5
          , V $ Circle 4 (2,3)
          , V $ Triangle
          , V $ Circle 1 (5,7)
          ]

We can rewrite resizeAll as follows:

resizeAllV :: AlterVariant Resizable cs => Float -> [V cs] -> [V cs]
resizeAllV f = fmap (alterVariant @Resizable (resize f))

It works on any list of variants as long as all the possible types in the heterogeneous list (i.e., types in cs) have an associated Resizable instance.

Let’s test it:

> shapes
[Square {squareTopLeft = (1.0,2.0), squareWidth = 5.0}
,Circle {circleRadius = 4.0, circleCenter = (2.0,3.0)}
,Circle {circleRadius = 1.0, circleCenter = (5.0,7.0)}]

> resizeAllV 2 shapes
[Square {squareTopLeft = (1.0,2.0), squareWidth = 10.0}
,Circle {circleRadius = 8.0, circleCenter = (2.0,3.0)}
,Circle {circleRadius = 2.0, circleCenter = (5.0,7.0)}]

We can check that type-checking works as expected by using this function on shapes2:

> resizeAllV 2 shapes2

<interactive>:3:1: error:
    • No instance for (Resizable Triangle)
        arising from a use of ‘resizeAllV’
    • In the expression: resizeAllV 2 shapes2
      In an equation for ‘it’: it = resizeAllV 2 shapes2

Indeed the type of shapes2 indicates that it may contain triangles but Triangle has no Resizable instance!

Another approach is to create an instance of Resizable for any Variant cs as long as every type in cs has a Resizable instance. It is defined recursively as follows:

-- this instance is never used but it is necessary for the type-checker
instance Resizable (V '[]) where
   {-# INLINE resize #-}
   resize = undefined

instance (Resizable (V xs), Resizable x) => Resizable (V (x ': xs)) where
   {-# INLINE resize #-}
   resize f v = case popVariantHead v of
      Right x -> toVariantHead (resize f x)
      Left xs -> toVariantTail (resize f xs)

We can now use the original resizeAll function:

resizeAll :: Resizable a => Float -> [a] -> [a]
resizeAll f = fmap (resize f)

> resizeAll 3 shapes
[Square {squareTopLeft = (1.0,2.0), squareWidth = 15.0}
,Circle {circleRadius = 12.0, circleCenter = (2.0,3.0)}
,Circle {circleRadius = 3.0, circleCenter = (5.0,7.0)}
]

> resizeAll 3 shapes2

<interactive>:66:1: error:
    • No instance for (Resizable Triangle)
        arising from a use of ‘resizeAll’
    • In the expression: resizeAll 3 shapes2
      In an equation for ‘it’: it = resizeAll 3 shapes2

We use this mechanism to provide Eq, Ord and Show instances. That’s how GHCI shows the variant contents.

We can easily compose functions by “composing” the constraints on the collection types:

import Data.Foldable (traverse_)

-- print an heterogeneous collection
printAll :: TraverseVariant Show cs IO => [V cs] -> IO ()
printAll = traverse_ (traverseVariant_ @Show print)

resizeThenPrint :: (AlterVariant Resizable cs, TraverseVariant Show cs IO)
                  => Float -> [V cs] -> IO ()
resizeThenPrint f = printAll . resizeAllV f

> resizeThenPrint 2 shapes
Square {squareTopLeft = (1.0,2.0), squareWidth = 10.0}
Circle {circleRadius = 8.0, circleCenter = (2.0,3.0)}
Circle {circleRadius = 2.0, circleCenter = (5.0,7.0)}

We can map a function to modify only values having some given type:

translateCircle :: Float -> Float -> Circle -> Circle
translateCircle fx fy (Circle r (x,y)) = Circle r (fx+x,fy+y)

> fmap (mapVariant (translateCircle 50 70)) shapes
[Square {squareTopLeft = (1.0,2.0), squareWidth = 5.0}
,Circle {circleRadius = 4.0, circleCenter = (52.0,73.0)}
,Circle {circleRadius = 1.0, circleCenter = (55.0,77.0)}
]

We can also map to a Variant and fold the result:

transformCircle :: Circle -> V '[Square,Triangle]
transformCircle (Circle r xy)
   | r > 3     = V (Square xy r)
   | otherwise = V Triangle

> shapes
[Square {squareTopLeft = (1.0,2.0), squareWidth = 5.0}
,Circle {circleRadius = 4.0, circleCenter = (2.0,3.0)}
,Circle {circleRadius = 1.0, circleCenter = (5.0,7.0)}]

> fmap (foldMapVariant transformCircle) shapes
[Square {squareTopLeft = (1.0,2.0), squareWidth = 5.0}
,Square {squareTopLeft = (2.0,3.0), squareWidth = 4.0}
,Triangle]

> :t fmap (foldMapVariant transformCircle) shapes
fmap (foldMapVariant transformCircle) shapes
  :: [V '[Square, Triangle, Square]]

As you can see, it is possible to have the same type more than once in a Variant. It is useful if you “select” types by index, e.g., fromVariantAt @2 v (cf. *At functions in Haddock).

You can also decide to keep a single entry per type with nubVariant:

> :t fmap (nubVariant . foldMapVariant transformCircle) shapes
fmap (nubVariant . foldMapVariant transformCircle) shapes
  :: [V '[Square, Triangle]]

You can also extend or reorder the type list of a Variant with liftVariant:

> shapes3 = fmap liftVariant shapes :: [V '[Square,Int,Circle,String]]

> shapes3 ++ [V "Hey!", V (10 :: Int)]
[Square {squareTopLeft = (1.0,2.0), squareWidth = 5.0}
,Circle {circleRadius = 4.0, circleCenter = (2.0,3.0)}
,Circle {circleRadius = 1.0, circleCenter = (5.0,7.0)}
,"Hey!"
,10
]

Obviously, we are not allowed to remove a type from the list (considered as a set):

> fmap liftVariant shapes :: [V '[Square,Int,String]]

<interactive>:112:6: error:
    • `Circle' is not a member of '[Square, Int, String]
    • In the first argument of ‘fmap’, namely ‘liftVariant’
      In the expression:
          fmap liftVariant shapes :: [V '[Square, Int, String]]
      In an equation for ‘it’:
          it = fmap liftVariant shapes :: [V '[Square, Int, String]]

<interactive>:112:6: error:
    • '[Circle, Square]
      is not a subset of
      '[Square, Int, String]
    • In the first argument of ‘fmap’, namely ‘liftVariant’
      In the expression:
          fmap liftVariant shapes :: [V '[Square, Int, String]]
      In an equation for ‘it’:
          it = fmap liftVariant shapes :: [V '[Square, Int, String]]

Let’s try to extract some shapes from the collection depending on their type:

import Data.Maybe (mapMaybe)

-- extract values having type "c" from the collection ("cs" must contain "c")
filterColl :: forall c cs. Popable c cs => [V cs] -> [c]
filterColl = mapMaybe (fromVariant @c)

> filterColl @Circle shapes
[Circle {circleRadius = 4.0, circleCenter = (2.0,3.0)}
,Circle {circleRadius = 1.0, circleCenter = (5.0,7.0)}]

> filterColl @Square shapes
[Square {squareTopLeft = (1.0,2.0), squareWidth = 5.0}]

> filterColl @Triangle shapes

<interactive>:44:1: error:
    • `Triangle' is not a member of '[Circle, Square]
    • In the expression: filterColl @Triangle shapes
      In an equation for ‘it’: it = filterColl @Triangle shapes

Indeed no Triangle can be in the shapes collection as it is defined to hold only circles and squares.

We may want to write a generic function that filters any collection, even if we statically know that it can’t contain the given type. Let’s use fromVariantMaybe function and MaybePopable constraint for this:

-- extract values having type "c" from the collection ("cs" may not contain "c")
filterCollMaybe :: forall c cs. MaybePopable c cs => [V cs] -> [c]
filterCollMaybe = mapMaybe (fromVariantMaybe @c)

> filterCollMaybe @Triangle shapes
[]
> filterCollMaybe @Triangle shapes2
[Triangle]
> filterCollMaybe @Int shapes2
[]

Instead of keeping values having a given type, it may be more useful to filter out some types:

import Data.Either (partitionEithers)

removeAll :: forall c cs. MaybePopable c cs => [V cs] -> [V (Filter c cs)]
removeAll = fst . partitionEithers . fmap (popVariantMaybe @c)

> removeAll @Triangle shapes
[Square {squareTopLeft = (1.0,2.0), squareWidth = 5.0}
,Circle {circleRadius = 4.0, circleCenter = (2.0,3.0)}
,Circle {circleRadius = 1.0, circleCenter = (5.0,7.0)}
]

> removeAll @Triangle shapes2
[Square {squareTopLeft = (1.0,2.0), squareWidth = 5.0}
,Circle {circleRadius = 4.0, circleCenter = (2.0,3.0)}
,Circle {circleRadius = 1.0, circleCenter = (5.0,7.0)}
]

> :t removeAll @Triangle shapes2
removeAll @Triangle shapes2 :: [V '[Circle, Square]] -- no more Triangle

> removeAll @Circle (removeAll @Triangle shapes2)
[Square {squareTopLeft = (1.0,2.0), squareWidth = 5.0}]

Now let’s try some pattern-matching:

matchShapes :: (Popable Circle cs
               ,Popable Square cs
               ,MaybePopable Triangle cs
               ) => V cs -> String
matchShapes = \case
   V (Circle r _)         -> "circle radius: " ++ show r
   V (Square _ w)         -> "square width: " ++ show w
   VMaybe (x :: Triangle) -> "triangle: " ++ show x
   _                      -> "unknown shape"

> fmap matchShapes shapes
["square width: 5.0"
,"circle radius: 4.0"
,"circle radius: 1.0"
]

> fmap matchShapes shapes2
["square width: 5.0"
,"circle radius: 4.0"
,"triangle: Triangle"
,"circle radius: 1.0"
]

• V pattern synonym and Popable constraint are used for types that must be storable in the variant.
• VMaybe pattern synonym and MaybePopable constraint are used for types that may not be storable in the variant.

Sadly when we use pattern matching on a Variant, the compiler can’t detect if we match all the possible types in the Variant. Hence we need a failover wildcard match to avoid a “non-exhaustive pattern match” warning:

matchShapes' :: V '[Circle,Square] -> String
matchShapes' = \case
   V (Circle r _)         -> "circle radius: " ++ show r
   V (Square _ w)         -> "square width: " ++ show w
   _                      -> "unknown shape" -- required useless failover match

Maybe we could circumvent this with a compiler plugin (left as an exercise for the motivated reader).

Conclusion

I hope this introduction to the use of the Haskus.Utils.Variant type has been interesting. You can find it in the haskus-utils package either on GitHub or on Hackage. Feedback is welcome!

There was already a brief description of the use of the Variant data type to handle heterogeneous collections on this blog buried at the end of this previous post about control-flow. Nevertheless, after reading this recent post about using existentials, Typeable, Dynamic and the like to manage heterogeneous collections in Haskell, I figured it deserved its own post. It led to a major API update and to a new release of haskus-utils package: I should write posts more often.

The first drafts of this post started with a presentation of the usual approaches to heterogeneous collections (i.e., the state-of-the-art). It was a bit boring and many readers would probably have left without reaching the Variant part, hence the state-of-the-art is now presented in the annexe of this post.

Annexe: alternatives and comparison

The wiki lists several solutions to the “heterogeneous collections” problem:

• Tuples
• Algebraic datatypes
• Data.Dynamic
• Existential types
• HLists

To which we should add:

• recursive GADT
• Variant

Let’s compare them!

Tuples

We don’t want a statically fixed length collection, hence we exclude tuples.

Algebraic datatypes

The idea is to create a wrapper Shape datatype:

data Shape
   = ShapeCircle Circle
   | ShapeSquare Square

We can write an instance:

instance Resizable Shape where
   resize f (ShapeCircle c) = ShapeCircle (resize f c)
   resize f (ShapeSquare s) = ShapeSquare (resize f s)

This is not easily extensible: if we want to support a new shape (e.g., a triangle), we have to modify both the Shape datatype and its Resizable instance:

data Shape
   = ShapeCircle Circle
   | ShapeSquare Square
   | ShapeTriangle Triangle

instance Resizable Shape where
   ..
   resize (ShapeTriangle ...) = ...

Now suppose that for some reason we want to statically ensure that a collection only contains shapes with a central symmetry (e.g., squares and circles but not triangles). The only way is to have different shape types for the different constraints on the shapes:

data CentralSymShape
   = CentralSymCircle Circle
   | CentralSymSquare Square

fooAll :: [CentralSymShape] -> [CentralSymShape]

But now we can’t resize elements of this collection anymore, so we need:

instance Resizable CentralSymShape where
   ...

Now adding a new shape is even more cumbersome because we potentially have to modify several datatypes and their instances.

Existential types

We can use existential types to embed some type-classes alongside some values. For instance we could rewrite our previous datatype as follows:

data Shape = forall a. Resizable a => Shape a

instance Resizable Shape where
   resize f (Shape a) = Shape (resize f a)

The problem is that we have lost the knowledge of the actual types in the collection: we only know that they are Resizable. For instance we can’t easily write a function extracting the circles:

extractCircles :: [Shape] -> [Circle]
extractCircles xs = ???

In addition we can’t easily reuse collections. Suppose we also have a type-class Drawable:

class Drawable a where
   draw :: a -> IO ()

instance Drawable Circle where ...
instance Drawable Square where ...
instance Drawable Triangle where ...

We could write a drawAll function by using the same technique:

data DrawableShape = forall a. Drawable a => DrawableShape a

instance Drawable DrawableShape where
   draw (DrawableShape a) = draw a

drawAll :: [DrawableShape] -> IO ()
drawAll = traverse_ draw

Now if we want to resize and then to draw a collection of shapes:

resizeAndDraw :: Float -> [?] -> IO ()
resizeAndDraw f = drawAll . resizeAll f

We need to introduce another wrapper datatype with its boilerplate:

data ResizableDrawableShape = forall a. (Drawable a, Resizable a) => ResizableDrawableShape a

instance Drawable ResizableDrawableShape where
   draw (ResizableDrawableShape a) = draw a

instance Resizable ResizableDrawableShape where
   resize f (ResizableDrawableShape a) = ResizableDrawableShape (resize f a)

resizeAndDraw :: Float -> [ResizableDrawableShape] -> IO ()
resizeAndDraw f = drawAll . resizeAll f

Hence we want to keep the knowledge of the types of the elements in the collection at runtime instead of using a different wrapper type per collection.

Data.Dynamic

Data.Dynamic can be used to wrap any value. It allows us to rewrite resizeAll as follows:

resizeAll :: Float -> [Dynamic] -> [Dynamic]
resizeAll f = fmap (resizeDyn f)

resizeDyn :: Float -> Dynamic -> Dynamic
resizeDyn f d =
   case fromDynamic d :: Maybe Circle of
      Just c  -> toDyn (resize f c)
      Nothing -> case fromDynamic d :: Maybe Square of
         Just s  -> toDyn (resize f s)
         Nothing -> error "Invalid shape"

The issue is that any datatype can be wrapped into a Dynamic, hence we lose some type safety. We want to restrict the types of the collection members using some constraints (Resizable, etc.).

HList

Instead of wrapping the elements of the collection, we could change the collection type to one that keeps track of the element types at compile time. For instance, HList. However the size of the “list” is fixed at compile time.